The use of the Gunning Fog Index to evaluate the readability of Polish and English drug leaflets in the context of Health Literacy challenges in Medical Linguistics: An exploratory study

Exploring the Role of Artificial Intelligence Chatbots in Preoperative Counseling for Head and Neck Cancer Surgery

OBJECTIVE

To evaluate the potential use of artificial intelligence (AI) chatbots, such as ChatGPT, in preoperative counseling for patients undergoing head and neck cancer surgery.

STUDY DESIGN

Cross-Sectional Survey Study.

SETTING

Single institution tertiary care center.

METHODS

ChatGPT was used to generate presurgical educational information including indications, risks, and recovery time for five common head and neck surgeries. Chatbot-generated information was compared with information gathered from a simple browser search (first publicly available website excluding scholarly articles). The accuracy of the information, readability, thoroughness, and number of errors were compared by five experienced head and neck surgeons in a blinded fashion. Each surgeon then chose a preference between the two information sources for each surgery.

RESULTS

With the exception of total word count, ChatGPT-generated pre-surgical information has similar readability, content of knowledge, accuracy, thoroughness, and numbers of medical errors when compared to publicly available websites. Additionally, ChatGPT was preferred 48% of the time by experienced head and neck surgeons.

CONCLUSION

Head and neck surgeons rated ChatGPT-generated and readily available online educational materials similarly. Further refinement in AI technology may soon open more avenues for patient counseling. Future investigations into the medical safety of AI counseling and exploring patients' perspectives would be of strong interest.

LEVEL OF EVIDENCE

N/A. Laryngoscope, 134:2757-2761, 2024.

Patient-Directed Vasectomy Information: How Readable Is It?

PURPOSE

To assess the quality and readability of online health information on vasectomy using validated readability and quality assessment tools.

MATERIALS AND METHODS

The top 50 search results for "vasectomy" on Google, Bing, and Yahoo were selected. Duplicate links, advertisements, blog posts, paid webpages, and information intended for healthcare providers were excluded. Flesch Reading Ease score, Flesch-Kincaid Grade level, Gunning Fog Index, and Simple Measure of Gobbledygook (SMOG) index were used to assess readability, with optimal readability level for online health information established as being at sixth grade reading level. DISCERN Instrument and JAMA Benchmark were used to assess the quality of selected webpages. Inter-assessment score correlation and results by webpage type were analyzed.

RESULTS

We analyzed 44 webpages, including 16 academic, 5 hospital-affiliated, 6 commercial, 13 non-profit health advocacy, and 4 uncategorized sources. The average readability of the evaluated webpages was at a 10th grade reading level as measured by the Flesch Kincaid Assessment tool, and an undergraduate reading level per the SMOG and Gunning Fog indices. Non-profit health advocacy webpages had the best reading level but still was not at the recommended level of grade 6 to 7. The overall DISCERN quality of the webpages was "fair", with non-profit health advocacy pages performing best.

CONCLUSIONS

The assessed webpages offer education on vasectomy in a language that is too complex for the general population to understand. Furthermore, several sources for online health information, such as non-profits, outperformed webpages by academic institutions. Increased healthcare collaboration and dedication to producing quality online patient resources is necessary to address these shortcomings and build trust among patients to increase utilization of vasectomy and decrease decisional regret.

Application of Artificial Intelligence to Patient-Targeted Health Information on Kidney Stone Disease

OBJECTIVE

The American Medical Association recommends health information to be written at a 6th grade level reading level. Our aim was to determine whether Artificial Intelligence can outperform the existing health information on kidney stone prevention and treatment.

METHODS

The top 50 search results for "Kidney Stone Prevention" and "Kidney Stone Treatment" on Google, Bing, and Yahoo were selected. Duplicate webpages, advertisements, pages intended for health professionals such as science articles, links to videos, paid subscription pages, and links nonrelated to kidney stone prevention and/or treatment were excluded. Included pages were categorized into academic, hospital-affiliated, commercial, nonprofit foundations, and other. Quality and readability of webpages were evaluated using validated tools, and the reading level was descriptively compared with ChatGPT generated health information on kidney stone prevention and treatment.

RESULTS

50 webpages on kidney stone prevention and 49 on stone treatment were included in this study. The reading level was determined to equate to that of a 10th to 12th grade student. Quality was measured as "fair" with no pages scoring "excellent" and only 20% receiving a "good" quality. There was no significant difference between pages from academic, hospital-affiliated, commercial, and nonprofit foundation publications. The text generated by ChatGPT was considerably easier to understand with readability levels measured as low as 5th grade.

CONCLUSIONS

The language used in existing information on kidney stone disease is of subpar quality and too complex to understand. Machine learning tools could aid in generating information that is comprehensible by the public.

Readability of Online Information for Parents Concerning Paediatric In-Toeing: An Analysis of the Most Popular Online Public Sources

Background Parents often access online resources to educate themselves on their child's condition. In-toeing, also referred to as pigeon toeing, is a common paediatric condition that has a variety of causes and is often a cause of concern for parents. With the increasing usage of the internet, parents of children with this condition may look to the web for answers. However, to be understood by the average adult, online health information must be written at an elementary school reading level. We hypothesised that currently available online resources regarding in-toeing would score poorly on objective measures of readability and understandability. Methods Patient education materials were identified via three commonly used online search engines (Google.com, Yahoo.com, and Bing.com). The terms "intoeing" and "pigeon toeing" were used for the search. From the top 50 search results, websites were included if directed at educating patients and their families regarding in-toeing. News articles, non-text material (video), industry websites, and articles not related to in-toeing were excluded. The readability was analysed using a specialised website www.readable.com to produce the following three scores: Gunning Fog Index (GFI), Flesch Reading Ease (FRE), and Flesch-Kincaid Grade (FKG). Understandability was calculated using the 19-point Patient Education Materials Assessment Tool (PEMAT). Results After removing duplicates, 84 unique websites were assessed for inclusion. A total of 48 websites and articles (57.14%) met the inclusion criteria. Of note, 23 articles out of 84 (27.38%) were excluded as they were intended for healthcare professionals. The means for the FRE, FKG, and GFI were 57.92 (±12.26), 7.92 (±1.91), and 9.35 (±2.36), respectively. Less than half of online resources had an FRE score at or higher than the recommended reading level for the general population. Mean understandability scores were 69.63% (±11.55%), with only 45.83% of articles being greater than the 70% requirement of adequate understandability. Conclusion Overall, online in-toeing educational materials scored poorly with respect to readability and understandability. Given the popularity of online resources in patient education, we should seek to improve this situation. Articles that are easier to read are thus more accessible to the general public and will aid in the shared decision-making process. Improved patient and parent satisfaction and overall standard of care can be expected.

Can ChatGPT Aid Clinicians in Educating Patients on the Surgical Management of Glaucoma?

PRCIS

ChatGPT can help health care providers automate the quality assessment of online health information, but it does not produce easier-to-understand responses compared with existing online health information.

PURPOSE

To compare the readability of ChatGPT-generated health information about glaucoma surgery to existing material online and to evaluate ChatGPT's ability to analyze the quality of information found online about glaucoma surgery.

METHODS

ChatGPT was asked to create patient handouts on glaucoma surgery using 7 independent prompts, aiming to generate sixth grade level reading material. Existing patient-targeted online health information about glaucoma surgery was selected from the top 50 search results of 3 search engines, excluding advertisements, blog posts, information intended for health professionals, irrelevant content, and duplicate links. Four validated tools were used to assess readability, and the readability of the ChatGPT-generated material was compared with the readability of existing online information. The DISCERN instrument was used for the quality assessment of online materials. The DISCERN instrument was also programmed to use ChatGPT to evaluate its ability to analyze quality. R software and descriptive statistics were used for data analysis.

RESULTS

Thirty-five webpages were included. There was no difference between the reading level of online webpages (12th grade) and the reading level of ChatGPT-generated responses (11th grade), despite the ChatGPT prompts asking for simple language and a sixth grade reading level. The quality of health content was "fair," with only 5 resources receiving an "excellent" score. ChatGPT scored the quality of health resources with high precision ( r =0.725).

CONCLUSIONS

Patient-targeted information on glaucoma surgery is beyond the reading level of the average patient, therefore at risk of not being understood, and is of subpar quality, per DISCERN tool scoring. ChatGPT did not generate documents at a lower reading level as prompted, but this tool can aid in automating the time-consuming and subjective process of quality assessment.

Assessing ChatGPT's ability to answer questions pertaining to erectile dysfunction: can our patients trust it?

Erectile dysfunction (ED) is a disorder that can cause distress and shame for men suffering from it. Men with ED will often turn to online support and chat groups to ask intimate questions about their health. ChatGPT is an artificial intelligence (AI)-based software that has been trained to engage in conversation with human input. We sought to assess the accuracy, readability, and reproducibility of ChatGPT's responses to frequently asked questions regarding the diagnosis, management, and care of patients with ED. Questions pertaining to ED were derived from clinic encounters with patients as well as online chat forums. These were entered into the free ChatGPT version 3.5 during the month of August 2023. Questions were asked on two separate days from unique accounts and computers to prevent the software from memorizing responses linked to a specific user. A total of 35 questions were asked. Outcomes measured were accuracy using grading from board certified urologists, readability with the Gunning Fog Index, and reproducibility by comparing responses between days. For epidemiology of disease, the percentage of responses that were graded as "comprehensive" or "correct but inadequate" was 100% across both days. There was fair reproducibility and median readability of 15.9 (IQR 2.5). For treatment and prevention, the percentage of responses that were graded as "comprehensive" or "correct but inadequate" was 78.9%. There was poor reproducibility of responses with a median readability of 14.5 (IQR 4.0). Risks of treatment and counseling both had 100% of questions graded as "comprehensive" or "correct but inadequate." The readability score for risks of treatment was median 13.9 (IQR 1.1) and for counseling median 13.8 (IQR 0.5), with good reproducibility for both question domains. ChatGPT provides accurate answers to common patient questions pertaining to ED, although its understanding of treatment options is incomplete and responses are at a reading level too advanced for the average patient.

Should I freeze my sperm?-readability and quality of health resources for sperm banking

Background

Sperm banking refers to the collection and storage of sperm cells for future use. Despite the recommendations of major medical societies, sperm banking is not discussed sufficiently with patients at risk of future fertility. Majority of Americans utilize the internet regarding health information. The aim of this study is to assess the reading level and the quality of online health information on sperm banking.

Methods

The top 50 search results from Google, Bing, and Yahoo were selected after searching for the term "sperm banking". Duplicate pages, advertisements, news and magazines, blog posts, videos, paid subscriptions, articles intended for health professionals, and non-related pages were excluded. Four validated readability and two quality assessment tools were used to score the text. Websites were divided into five categories: academic, hospital-affiliated, commercial, non-profit health advocacy, and non-categorized. Descriptive statistics, one sample t-test, and Pearson's correlation coefficient were used to analyze the data.

Results

Forty-one webpages were included. The mean Flesch Reading Ease Score (FRES) for all pages was 46.9/100 and the mean reading level was 11th grade, compared to the recommended 6th grade level, across various assessment tools. Utilizing the DISCERN Instrument, quality of online health information was fair. Seven percent of pages received a "good" quality score and no pages received a score of "excellent". On average, 1.5 out of 4 criteria categorized by the JAMA Benchmark, a validated quality assessment tool, were met. The hospital-affiliated webpages received the best reading scores and commercial pages received the highest quality scores.

Conclusions

Online health information on sperm banking available in English is of poor quality based on several quality assessment tools and at a reading level significantly higher than what is recommended. Further efforts are needed by providers and healthcare institutions to improve the quality of information available to patients.

Informed consent in dermatology: a narrative review

Informed consent is a legal and ethical obligation of healthcare providers, and there are several steps that must be followed for informed consent to be obtained. Numerous challenges exist to obtaining informed consent including gaps in health literacy, language barriers, impaired decision-making capacity, and inadequacy of informed consent forms. Dermatologists must understand the importance and process for obtaining informed consent to protect patients as well as avoid litigation. This narrative review provides an overview of the process of obtaining informed consent, insight into the top challenges that clinicians may face with suggested recommendations, and a brief review of litigation from improper consent among dermatologists.