An evaluation of the suitability, readability, and quality of online resources for the self-management of depression

The Evaluation of the Suitability, Quality, and Readability of Publicly Available Online Resources for the Self-Management of Fear of Cancer Recurrence

Cancer survivors often rely on the internet for health information, which has varying levels of readability, suitability, and quality. There is a need for high-quality online self-management resources for cancer survivors with fear of cancer recurrence (FCR). This study evaluated the readability, suitability, and quality of publicly available online FCR self-management resources. A Google search using FCR-related keywords identified freely available FCR self-management resources for cancer survivors in English. Resource readability (reading grade level), suitability, and quality were evaluated using relevant assessment tools. Descriptive statistics and cluster analysis identified resources with higher suitability and quality scores. Mean resource (n = 23) readability score was grade 11 (SD = 1.6, Range = 9-14). The mean suitability score was 56.0% (SD = 11.4%, Range = 31.0-76.3%), indicating average suitability and the mean quality score was 53% (SD = 11.7%, Range = 27-80%), indicating fair quality. A cluster of 15 (65%) resources with higher suitability and quality scores was identified. There were no significant associations between suitability or quality scores and the type of organisation that published the resources. Online FCR self-management resources varied in readability, suitability and quality. Resources with higher quality and suitability scores relative to other resources are identified for use by healthcare professionals and cancer survivors. Resources that are more culturally appropriate, with lower reading grade levels and detailed self-management strategies are needed.

Readability of publicly available mental health information: A systematic review

OBJECTIVE

Relatively little research has examined the health literacy demand of public communication (e.g. websites, pamphlets, and posters) about mental disorders. This study describes characteristics of existing research and summarizes what is known about the readability of such information.

METHODS

A search of eight data bases yielded 33 studies that met inclusion criteria. Studies were screened for relevance at each stage and then coded for key characteristics and appraised for quality by both authors working independently.

RESULTS

Grade-level readability scores as measured by the Flesch-Kincaid, SMOG, and other indices ranged from 6th to 17th grade. Most Flesch Reading Ease scores fell within the 30-50 range. Results of accessibility measures were inconsistent.

CONCLUSION

With the recommended readability level for health communication set at sixth grade, most publicly available information about mental disorders uses vocabulary and sentence structure that is too complex for the general public. Mixed evidence indicated that accessibility issues are also problematic.

PRACTICE IMPLICATIONS

Developers of health information need to decrease health literacy demand in written mental health information to ensure patients and their families may benefit from this information.

Evaluation of the usability, content, readability and cultural appropriateness of online alcohol and other drugs resources for Aboriginal and Torres Strait Islander Peoples in New South Wales, Australia

OBJECTIVES

This study aimed to analyse the usability, content, readability and cultural appropriateness of alcohol and other drugs (AODs) resources for Aboriginal and Torres Strait Islander Peoples in New South Wales (NSW), Australia.

OUTCOME MEASURES

The content of 30 AOD resources for Aboriginal and Torres Strait Islander Peoples was analysed according to the following criteria: general characteristics; elements of graphical design and written communication; thoroughness and content; readability (Flesch-Kincaid grade level (FKGL), Gunning Fog index (Fog), Simplified Measure of Gobbledygook and Flesch Reading Ease); and cultural appropriateness.

RESULTS

Most resources displayed good usability, depicted by the use of headings and subheadings (n=27), superior writing style (n=19), relevant visuals (n=19) and use of colour support (n=30). However, some resources used at least one professional jargon (n=13), and many did not provide any peer-reviewed references (n=22). During content analysis, 12 resources were categorised into the alcohol group and 18 resources in the other drugs group. Impact of alcohol during pregnancy and breast feeding (n=12) was the most common included topics in the resources related to alcohol, while the physical impact of drugs (n=15) was the most discussed topics among the other drugs group. Based on the FKGL readability score, 83% of resources met the recommended reading grade level of 6-8 by NSW Health. Many resources (n=21) met at least half of the cultural appropriateness elements of interest. However, less than one-third were developed in collaboration with the local community (n=9), used local terms (n=5), targeted the local community (n=3), included an Aboriginal voice (n=2) and addressed the underlying cause (n=1).

CONCLUSIONS

Many AOD resources are developed specifically for Aboriginal and Torres Strait Islander Peoples, but their usability, content and readability differed, and they were not culturally appropriate for all communities. Development of a standardised protocol for resource development is suggested.

An evaluation of the suitability, readability, quality, and usefulness of online resources for patients with cancer receiving immunotherapy

OBJECTIVE

To evaluate the suitability, readability, quality, and usefulness of available online resources about the side effects of immunotherapy.

METHODS

Resources were identified through online searches (Google, Microsoft Bing, and Yahoo search engines) and evaluated using the Suitability Assessment Materials (SAM), three readability formulae, the DISCERN tool (quality), and a checklist of symptom management strategies and self-management skills (usefulness). Hierarchical cluster analysis identified resources scoring highly for suitability and quality. Factors associated with the clusters were identified using Fischer's exact tests.

RESULTS

The 51 resources retained for evaluation were of low suitability and quality and exceeded recommended reading levels. Resources were lacking key features to support learning (e.g., illustrations) as well as information about the risks of treatments and the effect of treatments on quality of life. Few resources addressed key self-management skills of problem-solving and taking action. The cluster analysis identified "low" and "moderate" clusters for suitability and quality, and pamphlets and booklets were more likely to be in the moderate cluster as compared to webpages.

CONCLUSION

There is a lack of high-quality resources to support patients' learning to self-manage side effects of immunotherapy.

PRACTICE IMPLICATIONS

Results identify the best available online resources for self-managing side effects of immunotherapy and gaps for future work. Based on findings, a template for patient education materials was developed.

Development and validation of a questionnaire to test Chinese patients' knowledge of inflammatory bowel disease

A good understanding of a disease facilitates patient-centered management. We aimed to develop and validate a questionnaire to assess inflammatory bowel disease (IBD)-related knowledge and analyze the factors affecting patients' knowledge of IBD. We invited 15 experts to develop and modify an IBD knowledge questionnaires and 709 patients to test the reliability and validity of the questionnaires as well as analyze the factors related to the disease knowledge of patients with IBD. In internal consistency, Cronbach's α coefficients for the common items, ulcerative colitis (UC), and Crohn's disease (CD) knowledge questionnaires were 0.886, 0.89, and 0.886, respectively. In cross-item consistency, Spearman-Brown split coefficients of the common items, UC, and CD knowledge questionnaires were 0.843, 0.812, and 0.812, respectively. In time consistency, the test-retest reliability ICC was 0.862 (P < 0.001). The correlation between researcher scores, IBD-KNOW scores, and the original questionnaire scores was greater than 0.7 (P < 0.001). Multiple linear regression demonstrated that the factors, including disease type, age, body mass index, education level, income, treatment cost, duration of disease, and frequency of visits, affected the IBD patients' knowledge of the disease (P < 0.05). The IBD knowledge questionnaires had good reliability and validity and, therefore, can be used to assess patient knowledge of the disease.

Evaluation of quality, readability, suitability, and usefulness of online resources available to cancer survivors

PURPOSE

The aim of this study was to evaluate the quality, readability, suitability, and usefulness of resources publicly available to adult cancer survivors (aged 18 +) who have completed primary treatment.

METHODS

Resources were identified in July 2021 through Google. Search completeness was verified using Yahoo, Bing, and MedlinePlus. Retrieved resources were assessed for quality using the DISCERN, readability, suitability using the Suitability Assessment Measure (SAM), and usefulness based on a list of unmet needs and self-management skills derived from the literature. Descriptive analyses were conducted, and a cluster analysis identified the highest-scoring resources.

RESULTS

Forty-five resources were included. The mean DISCERN score was fair at 63.3% (SD 13.7%) with low-rated items being sources, publication date, and risks and mechanisms of treatment. The mean reading grade level was 11.19 (SD 1.61, range 8-16) with only one resource scoring an 8. The mean SAM score was in the adequate range at 48.2% (SD 10.6%), with graphics being the lowest-rated section. On average, included resources addressed 57.7% (SD 27.3%) of the unmet needs and 48.4% (SD 20.9%) of the self-management skills, the least addressed being problem-solving.

CONCLUSION

Quality and suitability were fair, whereas readability exceeded recommended levels. Only one resource had a superior score in both quality and suitability.

IMPLICATIONS FOR CANCER SURVIVORS

The most pressing need is to develop resources for cancer survivors that address their unmet needs and are accessible in terms of literacy. Study findings outline the highest-scoring resources currently available to survivors, families, and clinicians.

Can popular magazines educate people with depression? An investigation of articles' understandability, reliability, and actionability

Objective

to evaluate articles on depression published in popular magazines with respect to understandability, reliability, and actionability. To determine whether these articles can educate patients. To investigate whether the Clear Communication Index (CCI), developed to assess the quality of patient education materials produced by the medical sector, can be used to evaluate articles published in popular magazines.

Methods

The sample consists of 81 articles from 24 different Flemish and Dutch popular magazines. The articles were evaluated using the CCI. Correlational and t-test analyses were performed on the data.

Results

No more than one-fifth of all articles were found to be of quality. Significant positive correlations were found between actionability, reliability, and understandability. No significant differences were found between health magazines and other more generally oriented magazines.

Conclusion

Overall, our findings demonstrate the relative lack of power as patient educational materials of articles on depression published in popular magazines for people with low or average mental health literacy levels.

Innovation

Using the Clear Communication Index, the quality of Dutch medium popular magazine articles on depression are analyzed. The study design allowed for the comparison of different types of magazines. Health magazines do no score better than generally-focused magazines.

The Quality of Internet Websites for People Experiencing Psychosis: Pilot Expert Assessment

Background

Clinicians need to be able to assess the quality of the available information to aid clinical decision-making. The internet has become an important source of health information for consumers and their families.

Objective

This study aimed to rate the quality of websites with psychosis-related information (to provide clinicians with a basis for recommending material to guide clinical decision-making with consumers and their families), using a validated instrument as well as a purpose-developed checklist, and consider improvement in quality over a 4-year period.

Methods

Two measures of website quality were used: the DISCERN scale and the Psychosis Website Quality Checklist (PWQC). Terms related to psychosis, including “psychotic,” “psychosis,” “schizophrenia,” “delusion,” and “hallucination,” were entered into Google, and the first 25 results were analyzed. In total, 6 raters with varying health professional backgrounds were used to evaluate the websites across two time points: January-March 2014 and January-March 2018.

Results

Of the 25 websites rated, only the 6 highest ranked websites achieved a DISCERN score, indicating that they were of “good” quality (51-62 out of a possible 75), while the mean score of the websites (mean 43.96, SD 12.08) indicated an overall “fair” quality. The PWQC revealed that websites scored highly on “availability and usability” (mean 16.82, SD 3.96) but poorly on “credibility” (mean 20.99, SD 6.68), “currency” (mean 5.16, SD 2.62), and “breadth and accuracy” (mean 77.87, SD 23.20). Most sites lacked information about early intervention, recreational drug use and suicide risk, with little change in content over time. Stating an editorial or review process on the website (found in 56% of websites) was significantly associated with a higher quality score on both scales (the DISCERN scale, P=.002; the PWQC, P=.006).

Conclusions

The information on the internet available for clinicians to recommend to people affected by psychosis tended to be of “fair” quality. While higher-quality websites exist, it is generally not easy way to assess this on face value. Evidence of an editorial or review process was one indicator of website quality. While sites generally provided basic clinical information, most lacked material addressing weighing up risks and benefits of medication and alternatives, the role of coercive treatment and other more contentious issues. Insufficient emphasis is placed on detailed information on early intervention and importance of lifestyle modifications or how families and friends can contribute. These are likely to be the very answers that consumers and carers are seeking and this gap contributes to unmet needs among this group. We suggest that clinicians should be aware of what is available and where there are gaps.

An evaluation of the quality of online perinatal depression information

BACKGROUND

During the perinatal period (including pregnancy and up to 12 months after childbirth), expectant and new mothers are at an elevated risk of developing depression. Inadequate knowledge about perinatal depression and treatment options may contribute to the low help-seeking rates exhibited by perinatal people. The Internet can be an accessible source of information about perinatal depression; however, the quality of this information remains to be evaluated. The purpose of this study was to assess the quality of perinatal depression information websites.

METHODS

After review, 37 websites were included in our sample. To assess overall website quality, we rated websites based on their reading level (Simple Measure of Gobbledegook; SMOG), information quality (DISCERN), usability (Patient Education Materials Assessment Tool; PEMAT), and visual design (Visual Aesthetics of Website Inventory; VisAWI).

RESULTS

Websites often exceeded the National Institute of Health's recommended reading level of grades 6-8, with scores ranging from 6.8 to 13.5. Website information quality ratings ranged from 1.8 to 4.3 out of 5, with websites often containing insufficient information about treatment choices. Website usability ratings were negatively impacted by the lack of information summaries, visual aids, and tangible tools. Visual design ratings ranged from 3.2 to 6.6 out of 7, with a need for more creative design elements to enhance user engagement.

CONCLUSIONS

This study outlines the characteristics of high-quality perinatal depression information websites. Our findings illustrate that perinatal depression websites are not meeting the needs of users in terms of reading level, information quality, usability, and visual design. Our results may be helpful in guiding healthcare providers to reliable, evidence-based online resources for their perinatal patients.

Assessing the understandability, actionability, and quality of online resources for the self-management of bipolar disorder

Introduction

This study aims to assess the understandability, actionability, and quality of online resources for the self-management (SM) of bipolar spectrum disorders in adults.

Methods

An online search using Google, Bing, and Yahoo! search engines was conducted to identify resources for bipolar disorder. Those that were published in English, discussed at least 1 method directed at improving an SM task, and were within the first 25 nonadvertisement results for each search were included. Resources directed specifically at adolescents were excluded. Understandability and actionability of the online resources were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Quality of the online resources was evaluated using the DISCERN instrument. The number of SM tasks each resource discussed was also evaluated. Overall mean appropriateness was calculated by averaging the percentage scores of understandability, actionability, and quality.

Results

Fifty-two resources were included. The mean sample scores were 8.4 (SD, 2.1; range, 2-13; maximum, 15) for understandability, 2.2 (SD, 1.2; range, 0-4; maximum, 5) for actionability, and 46.1 (SD, 8.9; range, 30-57; maximum, 75) for quality. The overall mean appropriateness percentage was 53.5% (SD, 11.7%; range, 18%-77%), with a goal of at least 70%. Included resources addressed a mean of 7.1 tasks (SD, 2.5; range, 1-14; maximum, 20).

Discussion

Most online resources for the SM of bipolar disorder scored poorly for understandability and actionability based on PEMAT scores and had low to moderate scores for quality using the DISCERN instrument. Future online resources should be designed with the goal of increasing appropriateness for patients.

Predicting Writing Styles of Web-Based Materials for Children's Health Education Using the Selection of Semantic Features: Machine Learning Approach

BACKGROUND

Medical writing styles can have an impact on the understandability of health educational resources. Amid current web-based health information research, there is a dearth of research-based evidence that demonstrates what constitutes the best practice of the development of web-based health resources on children's health promotion and education.

OBJECTIVE

Using authoritative and highly influential web-based children's health educational resources from the Nemours Foundation, the largest not-for-profit organization promoting children's health and well-being, we aimed to develop machine learning algorithms to discriminate and predict the writing styles of health educational resources on children versus adult health promotion using a variety of health educational resources aimed at the general public.

METHODS

The selection of natural language features as predicator variables of algorithms went through initial automatic feature selection using ridge classifier, support vector machine, extreme gradient boost tree, and recursive feature elimination followed by revision by education experts. We compared algorithms using the automatically selected (n=19) and linguistically enhanced (n=20) feature sets, using the initial feature set (n=115) as the baseline.

RESULTS

Using five-fold cross-validation, compared with the baseline (115 features), the Gaussian Naive Bayes model (20 features) achieved statistically higher mean sensitivity (P=.02; 95% CI -0.016 to 0.1929), mean specificity (P=.02; 95% CI -0.016 to 0.199), mean area under the receiver operating characteristic curve (P=.02; 95% CI -0.007 to 0.140), and mean macro F1 (P=.006; 95% CI 0.016-0.167). The statistically improved performance of the final model (20 features) is in contrast to the statistically insignificant changes between the original feature set (n=115) and the automatically selected features (n=19): mean sensitivity (P=.13; 95% CI -0.1699 to 0.0681), mean specificity (P=.10; 95% CI -0.1389 to 0.4017), mean area under the receiver operating characteristic curve (P=.008; 95% CI 0.0059-0.1126), and mean macro F1 (P=.98; 95% CI -0.0555 to 0.0548). This demonstrates the importance and effectiveness of combining automatic feature selection and expert-based linguistic revision to develop the most effective machine learning algorithms from high-dimensional data sets.

CONCLUSIONS

We developed new evaluation tools for the discrimination and prediction of writing styles of web-based health resources for children's health education and promotion among parents and caregivers of children. User-adaptive automatic assessment of web-based health content holds great promise for distant and remote health education among young readers. Our study leveraged the precision and adaptability of machine learning algorithms and insights from health linguistics to help advance this significant yet understudied area of research.

Evaluation of online, publicly available cancer-related educational and self-management resources for symptom management

OBJECTIVE

The aims of this study were to evaluate the readability, suitability, and quality of publicly available online self-management (SM) resources for people with cancer.

METHODS

Resources were identified using two strategies: (1) a targeted search of 20 Canadian organizations and (2) a Google search. These were evaluated using the Suitability Assessment of Materials (SAM), the DISCERN tool for quality, and readability indices. The SM skills (e.g., problem-solving) and symptom management strategies addressed by each resource were also assessed. Descriptive and hierarchical cluster analyses were performed to identify resources of the highest suitability and quality as well as resource characteristics associated with higher quality and suitability.

RESULTS

A total of 92 resources were evaluated. The mean reading grade level for English resources was 10.29 (SD = 1.64, range of 7.05 to 15.09) and 12.62 for French resources (SD = 2.27, range of 10.12 to 15.65). The mean SAM score across the sample was 50.4% (SD = 10.6%), or 'adequate', and the mean DISCERN score was 61.1% (SD = 11.8%), or 'fair'. The cluster analysis indicated that 10 resources scored highly on both the SAM and the DISCERN. In total, 91 symptom management strategies were identified. On average, resources addressed 2.73 SM skills (SD = 1.58).

CONCLUSIONS

There is a need for plain language resources for people with lower reading ability and resources that incorporate more SM skills. Study findings will help healthcare professionals, patients, and their families identify optimal resources to address cancer-related symptoms.

Depression in cancer: quality assessment of online patient education resources

OBJECTIVE

Psychiatric comorbidities are common among cancer patients. However, little is known about the quality of online information regarding these conditions. This study uses a validated tool to systematically determine the strengths and limitations of websites addressing depression in cancer patients.

METHODS

The term "depression in cancer patients" was searched online using the search engines Google, Yippy, and Dogpile. A set of predetermined inclusion/exclusion criteria was applied to all search results, which yielded 48 websites for inclusion. A validated rating tool was used to score the websites based on the six domains of Affiliation, Accountability, Interactivity, Structure & Organization, Readability, and Content Quality. The results were analyzed using descriptive statistics.

RESULTS

Of the 48 websites evaluated, 50% were commercial. 63% of websites identified authorship, 54% cited reliable sources, 37% provided links, and 38% were updated within the last 2 years. 94% of websites featured a search engine and 60% had at least four structural tools. Average readability was at a grade 12.3 level using the Flesch-Kincaid scale and 11.3 using the Simple Measure of Gobbledygook Index. The most completely and accurately covered topics of depression were symptoms and treatment-83% and 73% respectively. Its prevention and prognosis were not covered by any of the websites.

CONCLUSIONS

A validated rating tool was applied to evaluate the quality of online information for depression in cancer patients. Website accountability was poor, readability was often at a level that is too difficult for the lay audience, and the topics of prevention and prognosis were seldom covered.

Assessing an educational booklet for promotion of healthy lifestyles in sedentary adults with cardiometabolic risk factors

OBJECTIVES

1) To evaluate and improve the suitability and readability of an educational booklet designed for use in an intervention aimed at promoting healthy lifestyles as a way to manage cardiometabolic risk factors in adults with low levels of physical activity and 2) to check its acceptability to real end users.

METHODS

A 7-phase methodology was used. The evaluation committee (9 experts, 6 potential end users) scored the booklet using the Suitability Assessment of Materials (SAM) and health education material (HEM) questionnaires on two occasions (original and revised version). Acceptability was also assessed by 75 real end users after a lifestyle intervention.

RESULTS

The SAM and HEM scores of the revised booklet were respectively 10% (SD = 9%) and 10% (SD = 7%) higher than the original booklet scores. The revised version attained "superior" scoring in all questionnaires: SAM, 87% (SD = 8%); HEM, 3.5 (SD = 0.4) out of 4; Acceptability, 3.7 (SD = 0.4) out of 4.

CONCLUSIONS

The suitability and readability of an educational booklet-to promote a healthy lifestyle-were improved following a systematic peer-review evaluation prior to its use in an intervention.

PRACTICE IMPLICATIONS

Improving the suitability and readability of educational resources can have a positive impact on patient knowledge and awareness, and hence on intervention effectiveness.