Evaluating the readability, understandability, and quality of online materials about chest pain in children

Patient education resources for oral mucositis: a google search and ChatGPT analysis

PURPOSE

Oral mucositis affects 90% of patients receiving chemotherapy or radiation for head and neck malignancies. Many patients use the internet to learn about their condition and treatments; however, the quality of online resources is not guaranteed. Our objective was to determine the most common Google searches related to "oral mucositis" and assess the quality and readability of available resources compared to ChatGPT-generated responses.

METHODS

Data related to Google searches for "oral mucositis" were analyzed. People Also Ask (PAA) questions (generated by Google) related to searches for "oral mucositis" were documented. Google resources were rated on quality, understandability, ease of reading, and reading grade level using the Journal of the American Medical Association benchmark criteria, Patient Education Materials Assessment Tool, Flesch Reading Ease Score, and Flesh-Kincaid Grade Level, respectively. ChatGPT-generated responses to the most popular PAA questions were rated using identical metrics.

RESULTS

Google search popularity for "oral mucositis" has significantly increased since 2004. 78% of the Google resources answered the associated PAA question, and 6% met the criteria for universal readability. 100% of the ChatGPT-generated responses answered the prompt, and 20% met the criteria for universal readability when asked to write for the appropriate audience.

CONCLUSION

Most resources provided by Google do not meet the criteria for universal readability. When prompted specifically, ChatGPT-generated responses were consistently more readable than Google resources. After verification of accuracy by healthcare professionals, ChatGPT could be a reasonable alternative to generate universally readable patient education resources.

Assessing parental comprehension of online resources on childhood pain

We aimed to examine the patient education materials (PEMs) on the internet about "Child Pain" in terms of readability, reliability, quality and content. For our observational study, a search was made on February 28, 2024, using the keywords "Child Pain," "Pediatric Pain," and "Children Pain" in the Google search engine. The readability of PEMs was assessed using computer-based readability formulas (Flesch Reading Ease Score [FRES], Flesch-Kincaid Grade Level [FKGL], Automated readability index (ARI), Gunning Fog [GFOG], Coleman-Liau score [CL], Linsear Write [LW], Simple Measure of Gobbledygook [SMOG]). The reliability and quality of websites were determined using the Journal of American Medical Association (JAMA) score, Global Quality Score (GQS), and DISCERN score. 96 PEM websites included in our study. We determined that the FRES was 64 (32-84), the FKGL was 8.24 (4.01-15.19), ARI was 8.95 (4.67-17.38), GFOG was 11 (7.1-19.2), CL was 10.1 (6.95-15.64), LW was 8.08 (3.94-19.0) and SMOG was 8.1 (4.98-13.93). The scores of readability formulas showed that, the readability level of PEMs was statistically higher than sixth-grade level with all formulas (P = .011 for FRES, P < .001 for GFOG, P < .001 for ARI, P < .001 for FKGL, P < .001 for CL and P < .001 for SMOG), except LW formula (P = .112). The websites had moderate-to-low reliability and quality. Health-related websites had the highest quality with JAMA score. We found a weak negative correlation between Blexb score and JAMA score (P = .013). Compared to the sixth-grade level recommended by the American Medical Association and the National Institutes of Health, the readability grade level of child pain-related internet-based PEMs is quite high. On the other hand, the reliability and quality of PEMs were determined as moderate-to-low. The low readability and quality of PEMs could cause an anxious parent and unnecessary hospital admissions. PEMs on issues threatening public health should be prepared with attention to the recommendations on readability.

Online health information on sinonasal inverted papillomas: An assessment on readability and quality

BACKGROUND/OBJECTIVES

Sinonasal inverted papilloma (IP) is a rare but serious diagnosis, with a paucity of patient-centred information regarding this condition. As more patients are seeking healthcare information online, the quality and comprehensibility of this information becomes ever more important. The aim of the study was to investigate the readability and quality of websites on inverted papilloma.

METHODS

The term IP and seven of its synonyms were inputted into the three of the most commonly used search engines in the English-speaking world (Google, Yahoo and Bing). The first 20 results returned for each search term were then screened with our exclusion criteria. The remaining websites were assessed for their readability using the using the Flesch Reading Ease Score (FRES) and average grade level (AGL). Quality was assessed using the DISCERN questionnaire.

RESULTS

Of the 480 websites returned using our search strategy, 410 were excluded using our screening criteria. Removal of duplicates from the remaining 70 websites left 14 for inclusion in the final analysis. The mean FRES score of the remaining websites was 30.5 ± 10 and the mean AGL was 15.2 ± 1.1, corresponding to a reading age of a 21-year-old. The median DISCERN score was 33.5 (30.5-36.5), a score which falls within the 'poor quality' range.

CONCLUSION

The readability and quality of online patient information on IP is far below the expected standard. Healthcare providers have a responsibility to direct patients to appropriate sources of information or consider producing new material should a lack of appropriate sources exist.

Rhytidectomy-Information Patients Seek and Where They Find Answers

Patients considering rhytidectomy often search for information online prior to in-office consultation. This study identifies the most searched queries regarding rhytidectomy and evaluates sources to which patients are directed. The search engine optimization tool Ahrefs was utilized to extract Google metadata on searches performed in the United States. Frequently asked questions were categorized by topic; websites were categorized by type. Journal of the American Medical Association (JAMA) benchmark criteria enabled information quality assessment. A total of 565 questions for three search phrases were extracted (265 "facelift," 265 "face lift," and 35 "rhytidectomy"). The majority of monthly searches in the facelift and face lift groups pertained to procedural cost, which was significantly higher than in the rhytidectomy group (52.9% and 50.7 vs. 0.0%, ANOVA p <0.001). The mean JAMA score for private practice sources (1.2 ± 0.42) was significantly lower than that of academic pages of (2.3 ± 1.9, p = 0.026) and commercial sources (3.0 ± 0.82, p = 0.008). The most popular destinations for rhytidectomy were California and Mexico (630 and 440 searches/month). Online searches for facelifts often revolve around the topic of cost and frequently direct patients to websites that provide inadequate information on authorship, attribution, disclosure, and currency.

How understandable are the patient education materials about flat foot on the Internet for parents?

Flat foot is a common reason for parents to visit orthopedic clinics. As the Internet has become an easy-search platform, parents often seek online educational materials before seeking out a professional. The aim of this study was to investigate the quality, readability, and understandability of such online materials for parents. An Internet search was performed for "flat foot" and "pes planus" using the Google search engine. The readability was evaluated using 6 different grading systems: Flesch Reading Ease Score, Flesch-Kincaid Grade Level, Simple Measure of Gobbledygook, Fry Readability score, Gunning Fog Index tests, and Automated Readability Index. The Patient Education Materials Assessment Tool test was used to assess the understandability. For quality assessment, the Journal of American Medical Association benchmark criteria and Health on the Net code were applied. One hundred nine websites were included and evaluated for readability, understandability, and quality. The mean readability grade for all websites was 10.5 ± 2.0. The mean Gunning Fog Index tests and Flesch-Kincaid Grade Level scores for all websites were 12.4 ± 2.2 and 9.7 ± 2.1 sequentially. The mean Coleman-Liau index score was 10.0 ± 1.5, and the average Fry Readability score was 9.9 ± 2.0. The automated readability index for all websites was 10.3 ± 2.5. The average Flesch Reading Ease score for all educational materials was 59.3 ± 10.1. The average Patient Education Materials Assessment Tool score for all educational materials was 81% (range, 70-87%). The mean Journal of American Medical Association benchmark criterion for all websites was 1.0, with a range from 1.0 and 2.0. Eighteen (16.5%) websites had Health on the Net certificates. Readability, understandability, and quality of patient education materials about flat feet on the Internet vary and are often worse than professional recommendations.

Patient education information material assessment criteria: A scoping review

BACKGROUND

Patient education information material (PEIM) is an essential component of patient education programs in increasing patients' ability to cope with their diseases. Therefore, it is essential to consider the criteria that will be used to prepare and evaluate these resources.

OBJECTIVE

This paper aims to identify these criteria and recognize the tools or methods used to evaluate them.

METHODS

National and international databases and indexing banks, including PubMed, Scopus, Web of Science, ProQuest, the Cochrane Library, Magiran, SID and ISC, were searched for this review. Original or review articles, theses, short surveys, and conference papers published between January 1990 and June 2022 were included.

RESULTS

Overall, 4688 documents were retrieved, of which 298 documents met the inclusion criteria. The criteria were grouped into 24 overarching criteria. The most frequently used criteria were readability, quality, suitability, comprehensibility and understandability.

CONCLUSION

This review has provided empirical evidence to identify criteria, tools, techniques or methods for developing or evaluating a PEIM. The authors suggest that developing a comprehensive tool based on these findings is critical for evaluating the overall efficiency of PEIM using effective criteria.

Assessment of Understandability and Actionability of YouTube Videos on Hemolytic Disease of the Newborn

Introduction With revolutions in Information Technology, information and misinformation are easier to be found online. YouTube is the largest and most commonly searched video content website in the world. It is assumed that, due to the coronavirus pandemic, most patients try to know about diseases through the internet and reduce the number of hospital exposures unless otherwise. In order to assess the understandability and actionability of such YouTube videos available freely online about the disease, Hemolytic disease of the newborn (HDN), this study was planned. Methods This is a cross-sectional study conducted with the first 160 videos available on May 14, 2021, with the search keyword "HDN" with a relevance filter and a duration of 4 to 20 minutes. The videos were further screened regarding the information content and language. These videos were assessed by three independent assessors using the patient educational materials assessment tool for audio-visual content. Results Out of the first 160 videos selected for screening, 58 videos were excluded due to a lack of content about the searched disease "HDN". Another 63 videos were excluded due to the language of instruction not being in English. Finally, 39 videos were assessed by three assessors. The understandability and actionability responses were checked for reliability and a Cronbach's alpha of 93.6% was found, indicating good data reliability. To reduce subjectivity, average scores of understandability and actionability were taken based on the scores of these three assessors. There were eight and 34 videos with average understandability and actionability scores of <70% respectively. The median average understandability and actionability scores were 84.4% and 50% respectively. There was a statistically significant difference between understandability and actionability scores with considerably lower actionability scores of YouTube videos on the disease, HDN (p<0.001). Conclusion There is a great need to include actionable information by content developers in videos. Most information available has adequate understandable content making it easier for the general public to know about the diseases. YouTube and similar social sites thus possibly are helping in the dissemination of information promoting awareness among the public in general and patients in particular.

Evidence-based health information about pulmonary embolism: Assessing the quality, usability and readability of online and offline patient information

Objective

Pulmonary embolism (PE) is the third most common cardiovascular disease worldwide. However, public awareness is considerably lower than for myocardial infarction or stroke. Patients suffering from PE complain about the lack of (understandable) information and express high informational needs. To uncover if reliable information is indeed scarce, this study evaluates the quantity and quality of existing patient information for tertiary prevention using an evidence-based health information paradigm.

Methods

We conducted a quantitative content analysis (n = 21 patient information brochures; n = 67 websites) evaluating content categories addressed, methodical quality, usability, and readability.

Results

Results show that there is not enough patient information material focusing on PE as a main topic. Existing patient information material is mostly incomplete, difficult to understand, and low in actionability as well as readability.

Conclusion

Our systematic analysis reveals the need for more high-quality patient information on PE as part of effective tertiary prevention.

Innovation

This is the first review analyzing content, methodical quality, readability, and usability of patient information on PE. The findings of this analysis are guiding the development of an innovative, evidence-based patient information on PE aiming to support patients' informational needs and their self-care behavior.

Assessing Communicative Effectiveness of Public Health Information in Chinese: Developing Automatic Decision Aids for International Health Professionals

Effective multilingual communication of authoritative health information plays an important role in helping to reduce health disparities and inequalities in developed and developing countries. Health information communication from the World Health Organization is governed by key principles including health information relevance, credibility, understandability, actionability, accessibility. Multilingual health information developed under these principles provide valuable benchmarks to assess the quality of health resources developed by local health authorities. In this paper, we developed machine learning classifiers for health professionals with or without Chinese proficiency to assess public-oriented health information in Chinese based on the definition of effective health communication by the WHO. We compared our optimized classifier (SVM_F5) with the state-of-art Chinese readability classifier (Chinese Readability Index Explorer CRIE 3.0), and classifiers adapted from established English readability formula, Gunning Fog Index, Automated Readability Index. Our optimized classifier achieved statistically significant higher area under the receiver operator curve (AUC of ROC), accuracy, sensitivity, and specificity than those of SVM using CRIE 3.0 features and SVM using linguistic features of Gunning Fog Index and Automated Readability Index (ARI). The statistically improved performance of our optimized classifier compared to that of SVM classifiers adapted from popular readability formula suggests that evaluation of health communication effectiveness as defined by the principles of the WHO is more complex than information readability assessment. Our SVM classifier validated on health information covering diverse topics (environmental health, infectious diseases, pregnancy, maternity care, non-communicable diseases, tobacco control) can aid effectively in the automatic assessment of original, translated Chinese public health information of whether they satisfy or not the current international standard of effective health communication as set by the WHO.

Google Search Analysis: What Do People Want to Know About Rhinoplasty and Where Do They Find the Answers?

Background: During online search queries, Google uses machine learning algorithms to provide frequently associated ("People Also Ask" [PAA]) questions with corresponding websites answering the question. We aimed to identify the most frequent questions about rhinoplasty asked online and the sources used to answer them. Materials and Methods: PAA questions were extracted for the terms "rhinoplasty," "nose surgery," and "nose job." Questions were categorized into specific topics. Websites were categorized by type and assessed for quality using Journal of the American Medical Association (JAMA) benchmark criteria. A search engine optimization tool determined search volume for individual questions and specific topics. Results: Internet searches for the PAA questions (n = 102) and associated websites were related to preoperative factors (46%), cost (35.7%), and recovery timeline (7.3%). Sources for the answers to PAA questions were single surgeon personal (39.3%) and medical practice (20.6%) websites. Conclusions: Surgeons may wish to emphasize specific patient education topics, including preoperative factors, cost, and recovery timeline, on their websites to address the most frequently sought-after information regarding rhinoplasty online.

Predicting Writing Styles of Web-Based Materials for Children's Health Education Using the Selection of Semantic Features: Machine Learning Approach

BACKGROUND

Medical writing styles can have an impact on the understandability of health educational resources. Amid current web-based health information research, there is a dearth of research-based evidence that demonstrates what constitutes the best practice of the development of web-based health resources on children's health promotion and education.

OBJECTIVE

Using authoritative and highly influential web-based children's health educational resources from the Nemours Foundation, the largest not-for-profit organization promoting children's health and well-being, we aimed to develop machine learning algorithms to discriminate and predict the writing styles of health educational resources on children versus adult health promotion using a variety of health educational resources aimed at the general public.

METHODS

The selection of natural language features as predicator variables of algorithms went through initial automatic feature selection using ridge classifier, support vector machine, extreme gradient boost tree, and recursive feature elimination followed by revision by education experts. We compared algorithms using the automatically selected (n=19) and linguistically enhanced (n=20) feature sets, using the initial feature set (n=115) as the baseline.

RESULTS

Using five-fold cross-validation, compared with the baseline (115 features), the Gaussian Naive Bayes model (20 features) achieved statistically higher mean sensitivity (P=.02; 95% CI -0.016 to 0.1929), mean specificity (P=.02; 95% CI -0.016 to 0.199), mean area under the receiver operating characteristic curve (P=.02; 95% CI -0.007 to 0.140), and mean macro F1 (P=.006; 95% CI 0.016-0.167). The statistically improved performance of the final model (20 features) is in contrast to the statistically insignificant changes between the original feature set (n=115) and the automatically selected features (n=19): mean sensitivity (P=.13; 95% CI -0.1699 to 0.0681), mean specificity (P=.10; 95% CI -0.1389 to 0.4017), mean area under the receiver operating characteristic curve (P=.008; 95% CI 0.0059-0.1126), and mean macro F1 (P=.98; 95% CI -0.0555 to 0.0548). This demonstrates the importance and effectiveness of combining automatic feature selection and expert-based linguistic revision to develop the most effective machine learning algorithms from high-dimensional data sets.

CONCLUSIONS

We developed new evaluation tools for the discrimination and prediction of writing styles of web-based health resources for children's health education and promotion among parents and caregivers of children. User-adaptive automatic assessment of web-based health content holds great promise for distant and remote health education among young readers. Our study leveraged the precision and adaptability of machine learning algorithms and insights from health linguistics to help advance this significant yet understudied area of research.