Tools for assessing readability and quality of health-related Web sites

An artificial intelligence language model improves readability of burns first aid information

AIMS

This study aimed to assess the potential of using an artificial intelligence (AI) large language model to improve the readability of burns first aid information.

METHODS

An AI language model (ChatGPT-3) was used to rewrite content from the top 50 English-language webpages containing burns first aid information to be understandable by an individual with the literacy level of an 11-year-old, as recommended by the American Medical Association and Health Education England. The assessment of readability was conducted using five validated tools.

RESULTS

In their original form, only 4% of the patient education materials (PEMs) met the target readability level across all tools. The median grade was 6.9 (SD=1.1). One-sample one-tailed t-test revealed that this was not significantly below the target (p = .31). After AI-modification, 18% of PEMs reached the target level using all tools, with a median grade of 6 (SD=0.9), which was significantly below the target level (p < .001). Once rewritten using AI, paired t-test demonstrated that all readability scores improved significantly (p < .001).

CONCLUSION

Utilising an AI language model proved an effective and viable method for enhancing readability of burns first aid information.

Dyslexia Articles Unboxed: Analyzing Their Readability Level

OBJECTIVE

Dyslexia is characterized by difficulties with fluent word recognition, decoding, or spelling, and it has been linked to family history. Given the impact of dyslexia on broad academic activities and well-being, ensuring that information about dyslexia is accessible to affected children and their families is vital. This study aims to assess the readability levels of dyslexia-related websites, with the hypothesis that such websites should be written at an appropriate readability level to accommodate those who may also have reading challenges.

METHODS

This study analyzes the readability of 50 web articles on dyslexia using 6 readability formulas: Flesch-Kincaid Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Score, Simple Measure of Gobbledygook (SMOG) Index, Coleman Liau Index, and Automated Readability Index. The search term "What is dyslexia" was used on Google. Each article was analyzed using the online calculation website WebFX. The readability goal for these websites was set at fifth grade, a level recommended for patients with reading challenges.

RESULTS

The study found that among the 50 websites, the lowest median readability score was 11.8 (corresponding to a 12th-grade level) on the SMOG Index, while the highest scores were 15.5 on both the Gunning Fog Score and the Coleman Liau Index (indicative of college-level readability). Almost none of the websites had scores below a fifth-grade level.

CONCLUSION

Most websites related to dyslexia are too complex. Tools such as readability metrics and sentence restructuring by AI can help make the information more accessible and understandable to the stakeholders.

Readability, accountability, and quality of burns first aid information available online

AIM

To assess the readability, accountability, and quality of burns first aid information available online.

METHODS

The top 50 English language webpages containing burns first aid information were compiled and categorised. Readability was measured using five validated tools. Accountability was assessed using the Journal of the American Medical Association (JAMA) benchmarks. Quality was evaluated using a scale based on previous literature.

RESULTS

Two (4%) webpages were judged to be at the target reading level using all tools. Median grade ranged from 4.6 to 9.6 (M = 6.9, SD = 1.1). One-sample one-tailed t-test determined that median grade was not significantly below the target grade of ≤ 6.9 (p = 0.314). Only seven (14%) webpages satisfied all the JAMA accountability benchmarks. No webpages fulfilled all 15 quality criteria. Mean quality score was 9.8 (SD = 2.4). Only 27 (54%) advised 20 min of cooling. One-way analysis of variance demonstrated that accountability was influenced by source (p = 0.01). Pearson's correlation coefficient revealed that accountability and quality had a positive correlation (r = 0.32, p = 0.02).

CONCLUSION

Much of the burns first aid information available online is written above the recommended reading level and fails to meet standards of accountability or quality.

Assessment of Quality and Readability of Online Patient-Centered Arabic Web-Based Knowledge About Apicoectomy

Background Endodontic microsurgery (apicectomy) can be considered in cases of persistent infection that is resistant to conventional root canal treatment. The aim of this study was to evaluate the quality and readability of the available online information regarding the apicectomy procedure in Arabic. Methods Online search on the three most commonly used websites (Google, Yahoo, and Bing) using one keyword. The first 100 websites from each search were analyzed for quality and readability using DISCERN instrument scores, the Journal of the American Medical Association (JAMA) benchmarks, the Health On the Net (HON) seal, Flesch Reading Ease Scores (FRES), Flesch-Kincaid Grade Level (FKGL), and the Simplified Measure of Gobbledygook (SMOG) Index. Results Searching using the Arabic translation for "root end resection surgery" revealed 349,900 websites. Following the inclusion criteria, 31 websites were selected and evaluated in this study. The selected websites belonged to either non-profit organizations or commercial websites. The quality of most of the selected websites received a moderate score (83.9%) using the DISCERN tool. None of the selected websites obtained the HON seal. Quality evaluation using the JAMA benchmarks revealed that currency was the most achieved item (45.2%), followed by authorship (22.6%). Evaluation of the readability of the selected websites using the FRES, FKGL, and SMOG showed that the included websites were considered readable. Conclusion Although the included websites were readable, the quality of the websites was moderate. There is an urgent need to create more trustworthy and readable websites explaining the different endodontic treatments.

Readability of online monkeypox patient education materials: Improved recognition of health literacy is needed for dissemination of infectious disease information

BACKGROUND

Health literacy is key to navigating the current global epidemic of misinformation and inaccuracy relating to healthcare. The American Medical Association (AMA) suggests health information should be written at the level of American sixth grade. With the monkeypox outbreak being declared a Public Health Emergency of International Concern (PHEIC) in July 2022, we sought to assess the readability of online patient education materials (PEMs) relating to monkeypox to see if they are at the target level of readability.

METHODS

A search was conducted on Google.com using the search term 'Monkeypox'. The top 50 English language webpages with patient education materials (PEMs) relating to monkeypox were compiled and categorised by country of publication and URL domain. Readability was assessed using five readability tools: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), and, Simple Measure of Gobbledygook Index (SMOG). Unpaired t-test for URL domain, and one-way ANOVA for country were performed to determine influence on readability.

RESULTS

Three of the five tools (FRES, GFI, CLI) identified no webpages that met the target readability score. The FKGL and SMOG tools identified one (2%) and two (4%) webpages respectively that met the target level. County and URL domain demonstrated no influence on readability.

CONCLUSION

Online PEMs relating to monkeypox are written above the recommended reading level. Based on the previously established effect of health literacy, this is likely exacerbating health inequalities. This study highlights the need for readability to be considered when publishing online PEMs.

Comparison of Readability Scores for Written Health Information Across Formulas Using Automated vs Manual Measures

This cross-sectional study examines the variability of readability scores across widely used online calculators.

Readability and Quality of Online Health Information Regarding Parathyroidectomy

Objective

Assessment of the readability and quality of online health information regarding parathyroidectomy.

Study Design

Cross-sectional analysis.

Setting

Websites providing patient-oriented health information regarding parathyroidectomy obtained via the Google search engine.

Methods

The top 75 Google search results for “parathyroidectomy,”“parathyroid surgery,” and “parathyroid gland removal” were reviewed. Websites were categorized by website type and country of origin. Readability was assessed by Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook. Website quality was assessed per JAMA benchmark criteria and the DISCERN instrument.

Results

A total of 74 unique websites were evaluated. The mean readability of the assessed websites exceeded the recommended sixth-grade reading level on the Flesch-Kincaid Grade Level and Simple Measure of Gobbledygook (P < .001). Readability did not vary significantly by website type. Websites originating from the United Kingdom were significantly more readable than those from the United States. The majority of assessed websites were of poor quality (n = 42, 56.8%) on assessment based on the DISCERN instrument. Quality varied significantly by website category on the JAMA benchmark criteria (P < .001) and DISCERN score (P = .049) with commercial websites receiving the highest scores. DISCERN score also varied significantly by country of origin (P = .036) with UK sites receiving highest mean DISCERN scores.

Conclusion

Online health information regarding parathyroidectomy is largely of poor quality and is poorly readable for many patients. Institutions utilizing well-defined guidelines for development of patient educational resources may provide online health information of greater quality and readability.

Forecasting Erroneous Neural Machine Translation of Disease Symptoms: Development of Bayesian Probabilistic Classifiers for Cross-Lingual Health Translation

BACKGROUND

Machine translation (MT) technologies have increasing applications in healthcare. Despite their convenience, cost-effectiveness, and constantly improved accuracy, research shows that the use of MT tools in medical or healthcare settings poses risks to vulnerable populations.

OBJECTIVES

We aimed to develop machine learning classifiers (MNB and RVM) to forecast nuanced yet significant MT errors of clinical symptoms in Chinese neural MT outputs.

METHODS

We screened human translations of MSD Manuals for information on self-diagnosis of infectious diseases and produced their matching neural MT outputs for subsequent pairwise quality assessment by trained bilingual health researchers. Different feature optimisation and normalisation techniques were used to identify the best feature set.

RESULTS

The RVM classifier using optimised, normalised (L₂ normalisation) semantic features achieved the highest sensitivity, specificity, AUC, and accuracy. MNB achieved similar high performance using the same optimised semantic feature set. The best probability threshold of the best performing RVM classifier was found at 0.6, with a very high positive likelihood ratio (LR+) of 27.82 (95% CI: 3.99, 193.76), and a low negative likelihood ratio (LR-) of 0.19 (95% CI: 0.08, 046), suggesting the high diagnostic utility of our model to predict the probabilities of erroneous MT of disease symptoms to help reverse potential inaccurate self-diagnosis of diseases among vulnerable people without adequate medical knowledge or an ability to ascertain the reliability of MT outputs.

CONCLUSION

Our study demonstrated the viability, flexibility, and efficiency of introducing machine learning models to help promote risk-aware use of MT technologies to achieve optimal, safer digital health outcomes for vulnerable people.

Help or hinder? An assessment of the accessibility, usability, reliability and readability of disability funding website information for Australian mental health consumers

This study aimed to assess the accessibility, usability, reliability and readability of those websites most likely encountered by Australian mental health consumers when using the internet to find information regarding the National Disability Insurance Scheme (NDIS). Websites were systematically identified with 127 deemed relevant for assessment in 2018. The LIDA instrument, the Simple Measure of Gobbledygook and Flesch-Kincaid Grade Level were used to evaluate the quality of information provided to mental health consumers. The study identified mediocre results for the accessibility, usability, reliability and readability of websites as they pertain to the needs of mental health consumers. Furthermore, it was identified that mental health support websites did not provide a more appropriate online experience for mental health consumers than general information websites, despite their focus on this demographic. These findings suggest a lack of understanding regarding the needs of mental health consumers and their experience of the online environment, which may in turn affect their access to information, agency and, ultimately, their uptake of the NDIS. The establishment of guidelines around enhancements to the online environment for mental health consumers would provide an experience that instils confidence, returns dignity and aids this group in realising their personal recovery journey.

The quality of non-alcoholic fatty liver disease information resources for patients on the internet in Korea

BACKGROUND/AIMS

Many patients uses the internet to obtain information about their diseases. However, there is increasing concern regarding the quality of internet information. Thus, we aimed to systematically evaluate the quality of websites containing educational information about non-alcoholic fatty liver disease (NAFLD) in Korea.

METHODS

Naver, Daum, and Google search engines were searched using the term "non-alcoholic fatty liver disease" in Korean. Two reviewers independently evaluated website quality using the quality evaluation instrument (QEI), which awarded websites scores for specific information on various aspects of NAFLD, as well as a five-point Likert scale (1-5), the DISCERN instrument, and a global quality scale (GQS).

RESULTS

Forty-seven websites met the inclusion criteria. We found that the quality of the internet information about NAFLD is generally poor. The mean QEI score with standard deviation was 10.31 ± 5.09 (range, 4 to 22), with only 17% of websites scoring higher than 10 points. The median GQS of the websites was 2.0, with no website achieving a score of 4 or 5. The QEI score was highly associated with the GQS score (r = 0.74, p < 0.01). For each DISCERN question from question 1 to question 15, the mean score was less than 3.

CONCLUSION

Overall, the internet health information for patients regarding NAFLD is poor and in need of much improvement. There is a need for institutional support, qualitative regulation of internet information, and development of an accreditation system to provide patients with internet health information of appropriate quality.

Detecting Symptom Errors in Neural Machine Translation of Patient Health Information on Depressive Disorders: Developing Interpretable Bayesian Machine Learning Classifiers

Background: Due to its convenience, wide availability, low usage cost, neural machine translation (NMT) has increasing applications in diverse clinical settings and web-based self-diagnosis of diseases. Given the developing nature of NMT tools, this can pose safety risks to multicultural communities with limited bilingual skills, low education, and low health literacy. Research is needed to scrutinise the reliability, credibility, usability of automatically translated patient health information. Objective: We aimed to develop high-performing Bayesian machine learning classifiers to assist clinical professionals and healthcare workers in assessing the quality and usability of NMT on depressive disorders. The tool did not require any prior knowledge from frontline health and medical professionals of the target language used by patients. Methods: We used Relevance Vector Machine (RVM) to increase generalisability and clinical interpretability of classifiers. It is a typical sparse Bayesian classifier less prone to overfitting with small training datasets. We optimised RVM by leveraging automatic recursive feature elimination and expert feature refinement from the perspective of health linguistics. We evaluated the diagnostic utility of the Bayesian classifier under different probability cut-offs in terms of sensitivity, specificity, positive and negative likelihood ratios against clinical thresholds for diagnostic tests. Finally, we illustrated interpretation of RVM tool in clinic using Bayes' nomogram. Results: After automatic and expert-based feature optimisation, the best-performing RVM classifier (RVM_DUFS12) gained the highest AUC (0.8872) among 52 competing models with distinct optimised, normalised features sets. It also had statistically higher sensitivity and specificity compared to other models. We evaluated the diagnostic utility of the best-performing model using Bayes' nomogram: it had a positive likelihood ratio (LR+) of 4.62 (95% C.I.: 2.53, 8.43), and the associated posterior probability (odds) was 83% (5.0) (95% C.I.: 73%, 90%), meaning that approximately 10 in 12 English texts with positive test are likely to contain information that would cause clinically significant conceptual errors if translated by Google; it had a negative likelihood ratio (LR-) of 0.18 (95% C.I.: 0.10,0.35) and associated posterior probability (odds) was 16% (0.2) (95% C.I: 10%, 27%), meaning that about 10 in 12 English texts with negative test can be safely translated using Google.

Web-Based Health Information Following the Renewal of the Cervical Screening Program in Australia: Evaluation of Readability, Understandability, and Credibility

BACKGROUND

Three main changes were implemented in the Australian National Cervical Screening Program (NCSP) in December 2017: an increase in the recommended age to start screening, extended screening intervals, and change from the Papanicolaou (Pap) test to primary human papillomavirus screening (cervical screening test). The internet is a readily accessible source of information to explain the reasons for these changes to the public. It is important that web-based health information about changes to national screening programs is accessible and understandable for the general population.

OBJECTIVE

This study aimed to evaluate Australian web-based resources that provide information about the changes to the cervical screening program.

METHODS

The term cervical screening was searched in 3 search engines. The first 10 relevant results across the first 3 pages of each search engine were selected. Overall, 2 authors independently evaluated each website for readability (Flesch Reading Ease [FRE], Flesch-Kincaid Grade Level, and Simple Measure of Gobbledygook [SMOG] index), quality of information (Patient Education Materials Assessment Tool [PEMAT] for printable materials), credibility (Journal of the American Medical Association [JAMA] benchmark criteria and presence of Health on the Net Foundation code of conduct [HONcode] certification), website design, and usability with 5 simulation questions to assess the relevance of information. A descriptive analysis was conducted for the readability measures, PEMAT, and the JAMA benchmark criteria.

RESULTS

Of the 49 websites identified in the search, 15 were eligible for inclusion. The consumer-focused websites were classed as fairly difficult to read (mean FRE score 51.8, SD 13.3). The highest FRE score (easiest to read) was 70.4 (Cancer Council Australia Cervical Screening Consumer Site), and the lowest FRE score (most difficult to read) was 33.0 (NCSP Clinical Guidelines). A total of 9 consumer-focused websites and 4 health care provider-focused websites met the recommended threshold (sixth to eighth grade; SMOG index) for readability. The mean PEMAT understandability scores were 87.7% (SD 6.0%) for consumer-focused websites and 64.9% (SD 13.8%) for health care provider-focused websites. The mean actionability scores were 58.1% (SD 19.1%) for consumer-focused websites and 36.7% (SD 11.0%) for health care provider-focused websites. Moreover, 9 consumer-focused and 3 health care provider-focused websites scored above 70% for understandability, and 2 consumer-focused websites had an actionability score above 70%. A total of 3 websites met all 4 of the JAMA benchmark criteria, and 2 websites displayed the HONcode.

CONCLUSIONS

It is important for women to have access to information that is at an appropriate reading level to better understand the implications of the changes to the cervical screening program. These findings can help health care providers direct their patients toward websites that provide information on cervical screening that is written at accessible reading levels and has high understandability.

Online information on Crohn's disease in Chinese: an evaluation of its quality and readability

To assess the quality and readability of patient-education information on Crohn's disease on the internet in China.

METHODS

Baidu and Sogou were chosen as search engines, and the top 58 webpages on Crohn's disease meeting the inclusion criteria were evaluated. After the data were screened, the quality of information was assessed by two specialists using a simplified DISCERN system. Different levels of official Chinese-teaching materials were used to build a readability model to predict the readability of these information.

RESULTS

Altogether 13 pieces of information were included in this study, all of which were derived from commercial webpages. The simplified DISCERN system found an excellent consistency in the Chinese literature, and Spearman's coefficient was 0.685, 0.556 and 0.607, respectively (P < 0.001), for publication reliability, treatment options and the total score. Most of the information lacked clear sources and conflicts of interest statements. The corresponding DISCERN scores were all below 2. In terms of treatment options, many webpages did not provide adequate information about the prognosis, possible side effects of treatment and risks of developing cancer. The scores of treatment-related DISCERN items were all below 2 as well. The webpages did not show enough readability. Altogether 84.6% of the information exceeded the middle-school readability level, and 69.2% exceeded the high-school level.

CONCLUSIONS

The quality of patient-education information on Crohn's disease on the internet in China is worrisome, with poor readability, which should be addressed by the academic community.

Readability and Understandability Analysis of Online Materials Related to Abdominal Aortic Aneurysm Repair

Background:

Patients commonly use online materials as a source of health information. Since poor health literacy has been shown to correlate with negative outcomes, it is recommended that patient-directed materials be written at a sixth-grade reading level. This study evaluates the readability and understandability of commonly accessed online materials pertaining to both endovascular and open repair of abdominal aortic aneurysm.

Methods:

Searches for “endovascular repair abdominal aortic aneurysm” and “open repair abdominal aortic aneurysm” were performed on both Google and Bing, and the top 10 websites from each search engine were identified. Relevant websites (total N = 28, endovascular n = 15, open n = 15, and 2 redundant sites) with patient-directed content were analyzed. Readability was assessed using 9 established methods, and understandability was assessed using the Patient Education Materials Assessment Tool scoring system.

Results:

The average reading grade level for all sites was 12.8. Endovascular sites averaged a reading grade level of 13.6 with a range from 11.5 to 15.6. Open-repair websites had a grade-level average of 12.1 with a range from 9.9 to 14.1. Readability was found to be inversely related to understandability, with a Pearson correlation coefficient of −0.551 ( P = .003). No website was written at or below the recommended sixth-grade reading level.

Conclusions:

Patient-directed online health information pertaining to open and endovascular repair of abdominal aortic aneurysm exceeds the recommended sixth-grade reading level. Increasing complexity of health literature correlates with poor understandability. Modifications such as shorter sentences, fewer words with more than 6 letters, and increasing usage of clear visual aids can increase readability and understandability.

Readability of online patient education materials for parents after a failed newborn hearing screen

OBJECTIVE

A cross-sectional study design was utilized to evaluate the readability of patient education materials on the newborn hearing screen from Google and major institutions.

METHODS

The top 55 websites from the Google search "failed newborn hearing screen" and websites from major institutions (the U.S. News & World Report ranked top 10 children's hospitals, the top 5 pediatric otolaryngology fellowships as ranked by Doximity Residency Navigator, the Centers for Disease and Control, the American Academy of Pediatrics and the American Academy of Otolaryngology-Head and Neck Surgery) were compiled. Text from each website was edited to remove extraneous text. Readability grade was calculated using the Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease Score, Gunning-Fog Index (GFI), Simple Measure of Gobbledygook (SMOG), Coleman-Liau Index, and Automated Readability Index. Intra- and inter-observer reliability were assessed.

RESULTS

26 websites from Google and 29 websites from major institutions were evaluated. From Google, provider-oriented websites (n = 2) were more difficult to read than patient-oriented websites (n = 24) with statistical significance for FKGL (p < 0.001), GFI (p < 0.013) and SMOG (p < 0.001). From the major institutions, more than half were at a reading level that exceeded the average American adult with an average FKGL 9.71 ± 2.69. Intra- and inter-observer reliability were both excellent with an intra-class correlation coefficient for each readability tool ≥0.950 (p < 0.001).

CONCLUSION

Online patient education materials about the newborn hearing screen may be too difficult for the average reader. Revisions to these materials and redirection to more readable online resources may be necessary to benefit a more inclusive patient population.

eHealth Literacy and General Interest in Using Online Health Information: A Survey Among Patients with Dental Diseases

Objectives

The aim of the study is to explore the eHealth literacy and general interest in using eHealth information among patients with dental diseases.

Methods

A total of 171 patients with dental diseases completed the survey including the eHEALS. The effect of participants' age, gender and education on eHealth literacy was assessed. Spearman’s correlation coefficient was also used to assess the correlation between the importance of access to health information and the usefulness of the internet for decision-making.

Results

The mean score of eHealth literacy in the participants was 30.55 (SD=4.069). The participants' age has significant effect on eHealth literacy level (t=3.573, P-value=0.002). Moreover, there was a significant correlation between the total score of eHealth literacy and the importance of access to eHealth information (r=0.33, n=171, P<0.s001). The difference in eHealth literacy in terms of educational background showed no statistically significant differences (F=1.179, P-value=0.322).

Discussion

The participants had a high level of eHealth literacy. Determining eHealth literacy among dental patients leads to a better understanding of their problems in health decision-making.

Conclusion

Dental institutions efforts should aim to raise awareness on online health information quality and to encourage patients to use evaluation tools, especially among low electronic health literate patients.

A Tool That Assesses the Evidence, Transparency, and Usability of Online Health Information: Development and Reliability Assessment

Background

The internet is commonly used by older adults to obtain health information and this trend has markedly increased in the past decade. However, studies illustrate that much of the available online health information is not informed by good quality evidence, developed in a transparent way, or easy to use. Furthermore, studies highlight that the general public lacks the skills necessary to distinguish between online products that are credible and trustworthy and those that are not. A number of tools have been developed to assess the evidence, transparency, and usability of online health information; however, many have not been assessed for reliability or ease of use.

Objective

The first objective of this study was to determine if a tool assessing the evidence, transparency, and usability of online health information exists that is easy and quick to use and has good reliability. No such tool was identified, so the second objective was to develop such a tool and assess it for reliability when used to assess online health information on topics of relevant to optimal aging.

Methods

An electronic database search was conducted between 2002 and 2012 to identify published papers describing tools that assessed the evidence, transparency, and usability of online health information. Papers were retained if the tool described was assessed for reliability, assessed the quality of evidence used to create online health information, and was quick and easy to use. When no one tool met expectations, a new instrument was developed and tested for reliability. Reliability between two raters was assessed using the intraclass correlation coefficient (ICC) for each item at two time points. SPSS Statistics 22 software was used for statistical analyses and a one-way random effects model was used to report the results. The overall ICC was assessed for the instrument as a whole in July 2015. The threshold for retaining items was ICC>0.60 (ie, “good” reliability).

Results

All tools identified that evaluated online health information were either too complex, took a long time to complete, had poor reliability, or had not undergone reliability assessment. A new instrument was developed and assessed for reliability in April 2014. Three items had an ICC<0.60 (ie, “good” reliability). One of these items was removed (“minimal scrolling”) and two were retained but reworded for clarity. Four new items were added that assessed the level of research evidence that informed the online health information and the tool was retested in July 2015. The total ICC score showed excellent agreement with both single measures (ICC=0.988; CI 0.982–0.992) and average measures (ICC=0.994; CI 0.991–0.996).

Conclusions

The results of this study suggest that this new tool is reliable for assessing the evidence, transparency, and usability of online health information that is relevant to optimal aging.

eHealth literacy and preferences for eHealth resources in parents of children with complex CHD

Introduction This study aimed to (a) examine eHealth literacy, beliefs, and behaviours in parents of children with complex CHD, and (b) identify parents' preferences for the content, format, features, and functions of eHealth resources for CHD. Materials and methods Families (n=198) of children born between 2008 and 2011 and diagnosed with CHD requiring surgery were mailed a survey assessing a range of variables including eHealth literacy, beliefs, and behaviours as well as preferences for the format, functions, features, and content of eHealth resources for CHD.

RESULTS

A total of 132 parents (83 mothers, 49 fathers) completed the survey (response rate: 50%). Mothers (96%) were more likely to access eHealth resources than fathers (83%, χ2=6.74, p=0.009). Despite high eHealth resource use, eHealth literacy was relatively low, with results demonstrating considerable and widespread gaps in awareness of, access to, and communication about eHealth resources. Over 50% of parents reported that decisions regarding their child's healthcare were influenced, to some extent, by web-based resources. Barriers to doctor-patient communication about eHealth included limited consultation time and concern about doctors' disapproval. Participants demonstrated a strong desire for "eHealth prescriptions" from their child's healthcare team, and perceived a wide range of eHealth topics as highly important, including treatment-related complications as well as physical, cognitive, and emotional development in children with CHD. Discussion Results suggest a need for stronger, more proactive partnerships between clinicians, researchers, educators, technologists, and patients and families to bring about meaningful innovations in the development and implementation of eHealth interventions in paediatric cardiology.

Readability Trends of Online Information by the American Academy of Otolaryngology—Head and Neck Surgery Foundation

Objective

Previous studies have shown that patient education materials published by the American Academy of Otolaryngology—Head and Neck Surgery Foundation may be too difficult for the average reader to understand. The purpose of this study was to determine if current educational materials show improvements in readability.

Study Design

Cross-sectional analysis.

Setting

The Patient Health Information section of the American Academy of Otolaryngology—Head and Neck Surgery Foundation website.

Subjects and Methods

All patient education articles were extracted in plain text. Webpage navigation, references, author information, appointment information, acknowledgments, and disclaimers were removed. Follow-up editing was also performed to remove paragraph breaks, colons, semicolons, numbers, percentages, and bullets. Readability grade was calculated with the Flesch-Kincaid Grade Level, Flesch Reading Ease, Gunning-Fog Index, Coleman-Liau Index, Automated Readability Index, and Simple Measure of Gobbledygook. Intra- and interobserver reliability were assessed.

Results

A total of 126 articles from 7 topics were analyzed. Readability levels across all 6 tools showed that the difficulty of patient education materials exceeded the abilities of an average American. As compared with previous studies, current educational materials by the American Academy of Otolaryngology—Head and Neck Surgery Foundation have shown a decrease in difficulty. Intra- and interobserver reliability were both excellent, with intraclass coefficients of 0.99 and 0.96, respectively.

Conclusion

Improvements in readability is an encouraging finding and one that is consistent with recent trends toward improved health literacy. Nevertheless, online patient educational material is still too difficult for the average reader. Revisions may be necessary for current materials to benefit a larger readership.

Online Health Information Regarding Male Infertility: An Evaluation of Readability, Suitability, and Quality

BACKGROUND

Many men lack knowledge about male infertility, and this may have consequences for their reproductive and general health. Men may prefer to seek health information online, but these sources of information vary in quality.

OBJECTIVE

The objective of this study is to determine if online sources of information regarding male infertility are readable, suitable, and of appropriate quality for Internet users in the general population.

METHODS

This study used a cross-sectional design to evaluate online sources resulting from search engine queries. The following categories of websites were considered: (1) Canadian fertility clinics, (2) North American organizations related to fertility, and (3) the first 20 results of Google searches using the terms "male infertility" and "male fertility preservation" set to the search locations worldwide, English Canada, and French Canada. Websites that met inclusion criteria (N=85) were assessed using readability indices, the Suitability Assessment of Materials (SAM), and the DISCERN tool. The associations between website affiliation (government, university/medical, non-profit organization, commercial/corporate, private practice) and Google placement to readability, suitability, and quality were also examined.

RESULTS

None of the sampled websites met recommended levels of readability. Across all websites, the mean SAM score for suitability was 45.37% (SD 11.21), or "adequate", while the DISCERN mean score for quality was 43.19 (SD 10.46) or "fair". Websites that placed higher in Google obtained a higher overall score for quality with an r (58) value of -.328 and a P value of .012, but this position was not related to readability or suitability. In addition, 20% of fertility clinic websites did not include fertility information for men.

CONCLUSIONS

There is a lack of high quality online sources of information on male fertility. Many websites target their information to women, or fail to meet established readability criteria for the general population. Since men may prefer to seek health information online, it is important that health care professionals develop high quality sources of information on male fertility for the general population.

Readability analysis of online resources related to lung cancer

BACKGROUND

Patients seeking health information commonly use the Internet as the first source for material. Studies show that well-informed patients have increased involvement, satisfaction, and healthcare outcomes. As one-third of Americans have only basic or below basic health literacy, the National Institutes of Health and American Medical Association recommend patient-directed health resources be written at a sixth-grade reading level. This study evaluates the readability of commonly accessed online resources on lung cancer.

METHODS

A search for "lung cancer" was performed using Google and Bing, and the top 10 websites were identified. Location services were disabled, and sponsored sites were excluded. Relevant articles (n = 109) with patient-directed content available directly from the main sites were downloaded. Readability was assessed using 10 established methods and analyzed with articles grouped by parent website.

RESULTS

The average reading grade level across all sites was 11.2, with a range from 8.8 (New Fog Count) to 12.2 (Simple Measure of Gobbledygook). The average Flesch Reading Ease score was 52, corresponding with fairly difficult to read text. The readability varied when compared by individual website, ranging in grade level from 9.2 to 15.2. Only 10 articles (9%) were written below a sixth-grade level and these tended to discuss simpler topics.

CONCLUSIONS

Patient-directed online information about lung cancer exceeds the recommended sixth-grade reading level. Readability varies between individual websites, allowing physicians to direct patients according to level of health literacy. Modifications to existing materials can significantly improve readability while maintaining content for patients with low health literacy.

Readability Assessment of Patient Information about Lymphedema and Its Treatment

BACKGROUND

Patient use of online resources for health information is increasing, and access to appropriately written information has been associated with improved patient satisfaction and overall outcomes. The American Medical Association and the National Institutes of Health recommend that patient materials be written at a sixth-grade reading level. In this study, the authors simulated a patient search of online educational content for lymphedema and evaluated readability.

METHODS

An online search for the term "lymphedema" was performed, and the first 12 hits were identified. User and location filters were disabled and sponsored results were excluded. Patient information from each site was downloaded and formatted into plain text. Readability was assessed using established tests: Coleman-Liau, Flesch-Kincaid, Flesch Reading Ease Index, FORCAST Readability Formula, Fry Graph, Gunning Fog Index, New Dale-Chall Formula, New Fog Count, Raygor Readability Estimate, and Simple Measure of Gobbledygook Readability Formula.

RESULTS

There were 152 patient articles downloaded; the overall mean reading level was 12.6. Individual website reading levels ranged from 9.4 (cancer.org) to 16.7 (wikipedia.org). There were 36 articles dedicated to conservative treatments for lymphedema; surgical treatment was mentioned in nine articles across four sites. The average reading level for conservative management was 12.7, compared with 15.6 for surgery (p < 0.001).

CONCLUSIONS

Patient information found through an Internet search for lymphedema is too difficult for many American adults to read. Websites queried had a range of readability, and surgeons should direct patients to sites appropriate for their level. There is limited information about surgical treatment available on the most popular sites; this information is significantly harder to read than sections on conservative measures.

Third Molars on the Internet: A Guide for Assessing Information Quality and Readability

BACKGROUND

Directing patients suffering from third molars (TMs) problems to high-quality online information is not only medically important, but also could enable better engagement in shared decision making.

OBJECTIVES

This study aimed to develop a scale that measures the scientific information quality (SIQ) for online information concerning wisdom tooth problems and to conduct a quality evaluation for online TMs resources. In addition, the study evaluated whether a specific piece of readability software (Readability Studio Professional 2012) might be reliable in measuring information comprehension, and explored predictors for the SIQ Scale.

METHODS

A cross-sectional sample of websites was retrieved using certain keywords and phrases such as "impacted wisdom tooth problems" using 3 popular search engines. The retrieved websites (n=150) were filtered. The retained 50 websites were evaluated to assess their characteristics, usability, accessibility, trust, readability, SIQ, and their credibility using DISCERN and Health on the Net Code (HoNCode).

RESULTS

Websites' mean scale scores varied significantly across website affiliation groups such as governmental, commercial, and treatment provider bodies. The SIQ Scale had a good internal consistency (alpha=.85) and was significantly correlated with DISCERN (r=.82, P<.01) and HoNCode (r=.38, P<.01). Less than 25% of websites had SIQ scores above 75%. The mean readability grade (10.3, SD 1.9) was above the recommended level, and was significantly correlated with the Scientific Information Comprehension Scale (r=.45. P<.01), which provides evidence for convergent validity. Website affiliation and DISCERN were significantly associated with SIQ (P<.01) and explained 76% of the SIQ variance.

CONCLUSION

The developed SIQ Scale was found to demonstrate reliability and initial validity. Website affiliation, DISCERN, and HoNCode were significant predictors for the quality of scientific information. The Readability Studio software estimates were associated with scientific information comprehensiveness measures.

How well do websites concerning children's anxiety answer parents' questions about treatment choices?

The goals of this study were to evaluate the quality of information concerning anxiety disorders in children that is available on the Internet and to evaluate changes in the quality of website information over time. The authors identified websites addressing child anxiety disorders (N = 26) using a Google search and recommendations from an expert in child anxiety. Each website was evaluated on the extent to which it addressed questions that parents consider important, the quality of information, and the reading level. All websites provided adequate information describing treatment options; however, fewer websites had information addressing many questions that are important to parents, including the duration of treatment, what happens when treatment stops, and the benefits and risks of various treatments. Many websites provided inadequate information on pharmacological treatment. Most websites were of moderate quality and had more difficult reading levels than is recommended. Five years after the initial assessment, authors re-analyzed the websites in order to investigate changes in content over time. The content of only six websites had been updated since the original analysis, the majority of which improved on the three aforementioned areas of evaluation. Websites could be strengthened by providing important information that would support parent decision-making.

Patient education for carpal tunnel syndrome: analysis of readability

BACKGROUND

The National Institutes of Health and American Medical Association recommend a sixth grade reading level for patient-directed content. This study aims to quantitatively evaluate the readability of the most commonly used resources for surgical treatment of carpal tunnel syndrome.

METHODS

A web search for "carpal tunnel surgery" was performed using an Internet search engine, and the 13 most popular sites were identified. Relevant, patient-directed articles immediately accessible from the main site were downloaded and formatted into plain text. A total of 102 articles were assessed for readability using ten established analyses: first overall, then by website for comparison.

RESULTS

Patient information about carpal tunnel surgery had an overall average reading level of 13.1. Secondary analysis by website revealed a range of mean readability from 10.8 (high school sophomore level) to 15.3 (university junior level). All sites exceeded the recommended sixth grade reading level.

CONCLUSIONS

Online patient resources for carpal tunnel surgery uniformly exceed the recommended reading level. These are too difficult to be understood by a large portion of American adults. A better understanding of readability may be useful in tailoring more appropriate resources for average patient literacy.

Readability Assessment of Online Patient Abdominoplasty Resources

BACKGROUND

Limited functional health literacy is recognized as an important contributor to health disparities in the United States. As internet access becomes more universal, there is increasing concern about whether patients with poor or marginal literacy can access understandable healthcare information. As such, the National Institutes of Health and American Medical Association recommend that patient information be written at a sixth grade level. This study identifies the most popular online resources for patient information about abdominoplasty and evaluates their readability in the context of average American literacy.

METHODS

The two largest internet search engines were queried for "tummy tuck surgery" to simulate a patient search in lay terms. The ten most popular sites common to both search engines were identified, and all relevant articles from the main sites were downloaded. Sponsored results were excluded. Readability analysis of the articles was performed using ten established tests.

RESULTS

Online information about abdominoplasty from the ten most popular publically available websites had an overall average readability of 12th grade. Mean reading grade level scores among tests were: Coleman-Liau 11.9, Flesch-Kincaid 11.4, FORCAST 11.1, Fry 13, Gunning Fog 13.5, New Dale-Chall 11.8, New Fog Count 9.9, Raygor Estimate 12, and SMOG 13.4; Flesch Reading Ease index score was 46.

CONCLUSIONS

Online patient resources about abdominoplasty are uniformly above the recommended target readability level and are likely too difficult for many patients to understand. A range of readability identified among websites could allow surgeons to guide patients to more appropriate resources for their literacy skills.

Does googling for preconception care result in information consistent with international guidelines: a comparison of information found by Italian women of childbearing age and health professionals

Background

Preconception counseling is effective in reducing the risk of adverse pregnancy outcomes. The Internet is commonly used by women and health professionals to search for health information. We compared the consistency of preconception information found on the Internet with the recommendations published by American Journal of Obstetrics and Gynecology (AJOG) simulating a web search by women of childbearing age and health professionals.

Methods

We reviewed websites resulting from a Google search performed using search strings selected by Italian women of childbearing age and health professionals. We investigated if retrieved information was consistent with AJOG recommendations for preconception care. Logistic regression was used to compare presence of consistent recommendations between women and health professionals.

Results

The highest frequency of correct recommendations was found for folic acid supplementation (39.4% of websites). Consistency of preconception information did not significantly differ between search strategies except for folic acid supplementation. “Communities and blogs” website category provided less frequently correct recommendations compared with “Medical/Public Agency" category (i.e. folic acid supplementation (aOR 0.254; CI 0.098-0.664; p = <0.01). Commercial links, found in 60% of websites, were associated with presence of correct recommendations excepting few items (i.e. physical exercise (aOR 1.127; CI 0.331-3.840; p = 0.848).

Conclusions

Preconception information found is poor and inaccurate regardless of the search is performed by women or health professionals. It is unlikely that information found on the web have any positive impact among women and health professionals in our setting. Strategies to improve preconception information on the web and education of health professionals for web searching of health information should be considered.

An assessment of the quality of information on stroke and speech and language difficulty web sites

Online healthcare information is used by patients, their carers and families (PCF) experiencing speech and language difficulties (SLD) after a stroke. This information may be of variable quality. Tools have been designed to assess the risk of poor information quality, evaluating both generic and specific concepts, yet none focuses on stroke. The readability of information is also an issue when communication disorders are under consideration. The study investigated the quality and readability of information on 51 web sites pertaining to SLD following a stroke. These were assessed using two generic evaluation tools (DISCERN and the HON Code), readability tests (Flesch Reading Ease and Flesch-Kincaid Grade Level) and a specifically designed Stroke Tool based on the reported information needs of stroke PCF experiencing SLD. The tools themselves were then evaluated for their feasibility, reliability and validity. In common with previous studies, the information quality of the selected web sites was found to be extremely variable with only 59% achieving a score of 50% or more using HON, only 37% using DISCERN, and 49% using the Stroke Tool. Readability is generally very poor with only 6% of web sites scoring below the recommended grade 7. The tools did not correlate well, suggesting that they measure different domains of quality. The Internet may be a valuable resource for stroke PCF; however, the variable quality of information means that cautious use is recommended. The readability of information is a more serious issue that needs addressing by web site developers.

Readability analysis of healthcare-oriented education resources from the American Academy of Facial Plastic and Reconstructive Surgery

OBJECTIVES/HYPOTHESIS

Deficient health literacy remains a widespread public issue. As such, the National Institutes of Health (NIH) recommends that all patient resources should be written around a sixth-grade level. The authors evaluate healthcare-oriented resources specified for patient use on the American Academy of Facial Plastic and Reconstructive Surgery (AAFPRS) Web site in order to identify potential areas of improvement and highlight those sections that may serve as paradigms for future revisions.

STUDY DESIGN

Descriptive and correlational design.

METHODS

Seventeen healthcare-oriented resources specifically for patients were downloaded in February 2012 from the American Academy of Facial Plastic and Reconstructive Surgery Web site. Readability assessments of each article were performed using Readability Studio Professional Edition Version 2012.1. These tests included the Flesch Reading Ease, Flesch-Kincaid Grade Level, SMOG Grading, Coleman-Liau Index, Gunning-Fog Index, the New Fog Count, the New Dale-Chall Readability Formula, FORCAST formula, Raygor Readability Estimate, and the Fry Graph.

RESULTS

Patient health education material found on the AAFPRS Web site has been found to be written at an average grade level of 12th grade using 10 different readability scales.

CONCLUSIONS

Modifications of the patient education section of the AAFPRS Web site can increase the readability of the literature, and allow greater comprehension among a wider audience.

How well does the Internet answer patients' questions about inflammatory bowel disease?

BACKGROUND

the Internet is an increasingly important source of health information.

OBJECTIVE

to assess how well common websites answered patients' questions regarding inflammatory bowel disease (IBD).

METHODS

thirty websites were identified and evaluated. Based on a previous survey of patient information needs, a comprehensive question list was developed in the three following areas: medical information (seven items), medical treatment (six items) and selfmanagement (eight items). The websites were evaluated for the amount of information they provided to answer each question using two standard measures of information quality - the DISCERN and the Ensuring Quality Information for Patients scales.

RESULTS

four particularly strong websites, scoring highest (on a scale from 1 to 5) in terms of IBD information, were the Crohn's and Colitis Foundation of America (mean information score 4.3), About.com (4.2), HealthCentral (3.8) and WebMD (3.8). These websites also scored well on the DISCERN and the Ensuring Quality Information for Patients quality scales. Most websites provided at least adequate information on common symptoms, complications, treatments and what is known (or not known) about the causes of IBD. However, many websites did not provide adequate information about prognosis, possible side effects of treatment and risks of developing cancer. Information regarding self-management was covered to a very limited extent.

CONCLUSION

websites could be strengthened by providing more of the information patients deem to be important, and by more clearly identifying sources of information and the date the information was updated. Most websites would benefit from more attention given to reducing the reading level and improving the organization of material.

Literacy assessment of family health history tools for public health prevention

OBJECTIVES

This study aimed to systematically identify and evaluate the readability and document complexity of currently available family history tools for the general public.

METHODS

Three steps were undertaken to identify family history tools for evaluation: (a) Internet searches, (b) expert consultation, and (c) literature searches. Tools identified were assessed for readability using the Simple Measure of Gobbledygook (SMOG) readability formula. The complexity of documents (i.e., forms collecting family history information) was assessed using the PMOSE/IKIRSCH document readability formula.

RESULTS

A total of 78 tools were identified, 47 of which met the criteria for inclusion. SMOG reading grade levels for multimedia-based tools ranged from 10.1 to 18.3, with an average score of 13.6. For print-based tools, SMOG ranged from 8.7 to 14.1, with an average score of 12.0. Document complexity ranged from very low complexity (level 1 proficiency) to high complexity (level 4 proficiency).

CONCLUSION

The majority of tools are written at a reading grade level that is beyond the 8th grade average reading level in the United States. The lack of family history tools that are easy to read or use may compromise their potential effectiveness in identifying individuals at increased risk for chronic diseases in the general population.

Readability analysis of patient information on the American Academy of Otolaryngology–Head and Neck Surgery website

Objective: To evaluate the readability of patient-oriented online health information (OHI) presented on the American Academy of Otolaryngology–Head and Neck Surgery (AAO–HNS) website.

Study Design: Review of the Flesch-Kincaid (FK) grade level for 104 articles on the AAO–HNS website.

Methods: The FK grade level for 104 articles was determined using the readability calculator available within Microsoft Office Word 2003. The interobserver reliability for the FK grade level was determined by calculating the intraclass correlation coefficient (ICC) for 52 entries.

Results: The average FK grade reading level of the articles was 10.8 (range 6.3-16.7; 95% CI, 10.4-11.2). Eighty-one percent of the articles were written at a ninth grade level or higher. The intraclass correlation was good (r = 0.83) for the 52 articles that were independently reviewed.

Conclusions: This analysis has shown that the average reading level for each article on the AAO–HNS site was higher than the recommended sixth grade reading level. Although the AAO–HNS site is written at a higher level than that suggested for the general public, it is important to realize that readability is just one consideration in the evaluation of OHI comprehension. Physicians need to be cognizant of their patients' ability to read and comprehend written information and tailor their educational material appropriately.