Readability of Online Hand Surgery Patient Educational Materials: Evaluating the Trend Since 2008

Readability of online patient education material for foregut surgery

INTRODUCTION

Health literacy is the ability of individuals to use basic health information and services to make well-informed decisions. Low health literacy among surgical patients has been associated with nonadherence to preoperative and/or discharge instructions as well as poor comprehension of surgery. It likely poses as a barrier to patients considering foregut surgery which requires an understanding of different treatment options and specific diet instructions. The objective of this study was to assess and compare the readability of online patient education materials (PEM) for foregut surgery.

METHODS

Using Google, the terms "anti-reflux surgery, "GERD surgery," and "foregut surgery" were searched and a total of 30 webpages from universities and national organizations were selected. The readability of the text was assessed with seven instruments: Flesch Reading Ease formula (FRE), Gunning Fog (GF), Flesch-Kincaid Grade Level (FKGL), Coleman Liau Index (CL), Simple Measure of Gobbledygook (SMOG), Automated Readability Index (ARI), and Linsear Write Formula (LWF). Mean readability scores were calculated with standard deviations. We performed a qualitative analysis gathering characteristics such as, type of information (preoperative or postoperative), organization, use of multimedia, inclusion of a version in another language.

RESULTS

The overall average readability of the top PEM for foregut surgery was 12th grade. There was only one resource at the recommended sixth grade reading level. Nearly half of PEM included some form of multimedia.

CONCLUSIONS

The American Medical Association and National Institute of Health have recommended that PEMs to be written at the 5th-6th grade level. The majority of online PEM for foregut surgery is above the recommended reading level. This may be a barrier for patients seeking foregut surgery. Surgeons should be aware of the potential gaps in understanding of their patients to help them make informed decisions and improve overall health outcomes.

Optimizing Readability of Patient-Facing Hand Surgery Education Materials Using Chat Generative Pretrained Transformer 3.5

PURPOSE

To address patient health literacy, the American Medical Association and the National Institutes of Health recommend that readability of patient education materials should not exceed an eighth grade reading level. However, patient-facing materials often remain above the recommended average reading level. Current online calculators provide readability scores; however, they lack the ability to provide text-specific feedback, which may streamline the process of simplifying patient materials. The purpose of this study was to evaluate Chat Generative Pretrained Transformer (ChatGPT) 3.5 as a tool for optimizing patient-facing hand surgery education materials through reading level analysis and simplification.

METHODS

The readability of 18 patient-facing hand surgery education materials was compared by a traditional online calculator for reading level and ChatGPT 3.5. The original excerpts were then entered into ChatGPT 3.5 and simplified by the artificial intelligence tool. The simplified excerpts were scored by the same calculators.

RESULTS

The readability scores for the original excerpts from the online calculator and ChatGPT 3.5 were similar. The simplified excerpts' scores were lower than the originals, with a mean of 7.28, less than the maximum recommended 8.

CONCLUSIONS

The use of ChatGPT 3.5 for the purpose of simplification and readability analysis of patient-facing hand surgery materials is efficient and may help facilitate the conveyance of important health information. ChatGPT 3.5 rendered readability scores comparable with traditional readability calculators, in addition to excerpt-specific feedback. It was also able to simplify materials to the recommended grade levels.

CLINICAL RELEVANCE

By confirming ChatGPT3.5's ability to assess and simplify patient education materials, this study offers a practical solution for potentially improving patient comprehension, engagement, and health outcomes in clinical settings.

A quality and readability comparison of artificial intelligence and popular health website education materials for common hand surgery procedures

INTRODUCTION

ChatGPT and its application in producing patient education materials for orthopedic hand disorders has not been extensively studied. This study evaluated the quality and readability of educational information pertaining to common hand surgeries from patient education websites and information produced by ChatGPT.

METHODS

Patient education information for four hand surgeries (carpal tunnel release, trigger finger release, Dupuytren's contracture, and ganglion cyst surgery) was extracted from ChatGPT (at a scientific and fourth-grade reading level), WebMD, and Mayo Clinic. In a blinded and randomized fashion, five fellowship-trained orthopaedic hand surgeons evaluated the quality of information using a modified DISCERN criteria. Readability and reading grade level were assessed using Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) equations.

RESULTS

The Mayo Clinic website scored higher in terms of quality for carpal tunnel release information (p = 0.004). WebMD scored higher for Dupuytren's contracture release (p < 0.001), ganglion cyst surgery (p = 0.003), and overall quality (p < 0.001). ChatGPT - 4th Grade Reading Level, ChatGPT - Scientific Reading Level, WebMD, and Mayo Clinic written materials on average exceeded recommended reading grade levels (4th-6th grade) by at least four grade levels (10th, 14th, 13th, and 11th grade, respectively).

CONCLUSIONS

ChatGPT provides inferior education materials compared to patient-friendly websites. When prompted to provide more easily read materials, ChatGPT generates less robust information compared to patient-friendly websites and does not adequately simplify the educational information. ChatGPT has potential to improve the quality and readability of patient education materials but currently, patient-friendly websites provide superior quality at similar reading comprehension levels.

ChatGPT-4 Can Help Hand Surgeons Communicate Better With Patients

The American Society for Surgery of the Hand and British Society for Surgery of the Hand produce patient-focused information above the sixth-grade readability recommended by the American Medical Association. To promote health equity, patient-focused content should be aimed at an appropriate level of health literacy. Artificial intelligence-driven large language models may be able to assist hand surgery societies in improving the readability of the information provided to patients. The readability was calculated for all the articles written in English on the American Society for Surgery of the Hand and British Society for Surgery of the Hand websites, in terms of seven of the commonest readability formulas. Chat Generative Pre-Trained Transformer version 4 (ChatGPT-4) was then asked to rewrite each article at a sixth-grade readability level. The readability for each response was calculated and compared with the unedited articles. Chat Generative Pre-Trained Transformer version 4 was able to improve the readability across all chosen readability formulas and was successful in achieving a mean sixth-grade readability level in terms of the Flesch Kincaid Grade Level and Simple Measure of Gobbledygook calculations. It increased the mean Flesch Reading Ease score, with higher scores representing more readable material. This study demonstrated that ChatGPT-4 can be used to improve the readability of patient-focused material in hand surgery. However, ChatGPT-4 is interested primarily in sounding natural, and not in seeking truth, and hence, each response must be evaluated by the surgeon to ensure that information accuracy is not being sacrificed for the sake of readability by this powerful tool.

Evaluation of Patient Education Materials From Large-Language Artificial Intelligence Models on Carpal Tunnel Release

BACKGROUND

ChatGPT, an artificial intelligence technology, has the potential to be a useful patient aid, though the accuracy and appropriateness of its responses and recommendations on common hand surgical pathologies and procedures must be understood. Comparing the sources referenced and characteristics of responses from ChatGPT and an established search engine (Google) on carpal tunnel surgery will allow for an understanding of the utility of ChatGPT for patient education.

METHODS

A Google search of "carpal tunnel release surgery" was performed and "frequently asked questions (FAQs)" were recorded with their answer and source. ChatGPT was then asked to provide answers to the Google FAQs. The FAQs were compared, and answer content was compared using word count, readability analyses, and content source.

RESULTS

There was 40% concordance among questions asked by the programs. Google answered each question with one source per answer, whereas ChatGPT's answers were created from two sources per answer. ChatGPT's answers were significantly longer than Google's and multiple readability analysis algorithms found ChatGPT responses to be statistically significantly more difficult to read and at a higher grade level than Google's. ChatGPT always recommended "contacting your surgeon."

CONCLUSION

A comparison of ChatGPT's responses to Google's FAQ responses revealed that ChatGPT's answers were more in-depth, from multiple sources, and from a higher proportion of academic Web sites. However, ChatGPT answers were found to be more difficult to understand. Further study is needed to understand if the differences in the responses between programs correlate to a difference in patient comprehension.

Assessing the Readability of Online English and Spanish Language Patient Education Resources Provided by the American Society of Plastic Surgeons, American Society of Aesthetic Plastic Surgeons, and American Society of Reconstructive Microsurgeons

INTRODUCTION

The National Institutes of Health recommends that patient education materials (PEMs) be written at the sixth grade level. However, PEMs online are still generally difficult to read. The usefulness of online PEMs depends on their comprehensibility.

OBJECTIVES

This study assessed the readability of PEMs from national Plastic and Reconstructive Surgery (PRS) organization websites.

METHODS

Patient education materials were collected from 3 prominent PRS organizations-the American Society of Plastic Surgeons (ASPS), American Society of Aesthetic Plastic Surgeons (ASAPS), and the American Society of Reconstructive Microsurgeons (ASRM). ASPS PEMs were organized into reconstructive and cosmetic groups, and then further subdivided into English and Spanish subgroups. ASAPS and ASRM PEMs provided cosmetic and reconstructive comparison groups to ASPS, respectively. Readability scores were generated using the Simple Measure of Gobbledygook (SMOG) and the Spanish SMOG scales.

RESULTS

Overall, all PEMs failed to meet readability guidelines. Within ASPS, Spanish PEMs were easier to read than English PEMs ( P < 0.001), and cosmetic PEMs were easier to read than reconstructive PEMs ( P < 0.05). There was no significant difference between ASPS cosmetic and ASAPS PEMs ( P = 0.36), nor between ASPS reconstructive and ASRM PEMs ( P = 0.65). ASAPS and ASRM did not have any Spanish PEMs, and 92% of all ASPS PEMs were in English.

CONCLUSION

Although PRS societies strive to better educate the public on the scope of PRS, PRS ranks lowly in public understanding of its role in patient care. In addition, Spanish language PEMs from the 3 PRS organizations are severely lacking. Addressing these concerns will make online patient resources more equitable for various patient populations.

Assessing the Readability of Online Patient Education Resources Related to Neophallus Reconstruction

INTRODUCTION

Online patient education materials (PEMs) often exceed the recommended 6th grade reading level. This can negatively affect transmasculine patients' understanding of treatment plans, increasing barriers to care and worsening health outcomes and patient satisfaction. This study assessed the readability of online English and Spanish PEMs regarding phalloplasty and urethroplasty.

METHODS

The English and Spanish terms for phalloplasty and urethroplasty were queried on Google. The first fifty results were grouped into institutional (government, medical school, teaching hospital), noninstitutional (private practice, news channel, blog, etc.), and academic (journal articles, book chapters) categories. Readability scores were generated using the Simple Measure of Gobbledygook and Spanish Simple Measure of Gobbledygook scales.

RESULTS

All PEMs exceeded recommended reading levels. For both procedures, English PEMs had an average reading level approximately of a university sophomore and Spanish PEMs had an average reading level approximately of a high school junior. For both procedures, English PEMs were harder to read than Spanish PEMs overall (P < 0.001) and when compared across the three categories between the two languages (P < 0.001). For Spanish urethroplasty PEMs, noninstitutional PEMs were more difficult to read than institutional PEMs (P < 0.05).

CONCLUSIONS

Online information for phalloplasty and urethroplasty should be revised and/or standardized materials should be created by trans-affirming health-care providers and national organizations in order to more fully educate the public and prospective patients prior to intervention. A well-informed patient population will improve patient decision-making and surgeon-patient communication, ultimately leading to better health outcomes.

Entering the Misinformation Age: Quality and Reliability of YouTube for Patient Information on Liposuction

Background: YouTube is currently the most popular online platform and is increasingly being utilized by patients as a resource on aesthetic surgery. Yet, its content is largely unregulated and this may result in dissemination of unreliable and inaccurate information. The objective of this study was to evaluate the quality and reliability of YouTube liposuction content available to potential patients. Methods: YouTube was screened using the keywords: "liposuction," "lipoplasty," and "body sculpting." The top 50 results for each term were screened for relevance. Videos which met the inclusion criteria were scored using the Global Quality Score (GQS) for educational value and the Journal of the American Medical Association (JAMA) criteria for video reliability. Educational value, reliability, video views, likes, dislikes, duration and publishing date were compared between authorship groups, high/low reliability, and high/low educational value. Results: A total of 150 videos were screened, of which 89 videos met the inclusion criteria. Overall, the videos had low reliability (mean JAMA score = 2.78, SD = 1.15) and low educational value (mean GQS score = 3.55, SD = 1.31). Videos uploaded by physicians accounted for 83.1% percent of included videos and had a higher mean educational value and reliability score than those by patients. Video views, likes, dislikes, comments, popularity, and length were significantly greater in videos with high reliability. Conclusions: To ensure liposuction-seeking patients are appropriately educated and informed, surgeons and their patients may benefit from an analysis of educational quality and reliability of such online content. Surgeons may wish to discuss online sources of information with patients.

Internet Search Trends for Common Hand Surgery Diagnoses

PURPOSE

The internet is a common resource for patients seeking health information. Trends in internet search interests for common hand surgery diagnoses and their seasonal variations have not been previously studied. The objectives of this study were (1) to describe the temporal trends in internet search interest for common hand surgery diagnoses in the recent five-year time period and (2) to assess seasonal variations in search term interest.

METHODS

An internet-based study of internet search term interest of 10 common hand surgery diagnoses was performed using Google Trends (Google, Inc., Mountain View, CA) from January 2017 to December 2021. The 10 diagnoses were "carpal tunnel syndrome," "trigger finger," "thumb arthritis," "ganglion cyst," "de Quervain's tenosynovitis," "lateral epicondylitis," "Dupuytren disease," "distal radius fracture," "finger fracture," and "scaphoid fracture." Analysis of variance (ANOVA) was used to assess for seasonal differences in search interest, and temporal trends were assessed using the two-tailed Mann-Kendall trend test.

RESULTS

During the study period, there was an increasing trend for search interest for "carpal tunnel syndrome," "trigger finger," "thumb arthritis," "Dupuytren disease," and "finger fracture," both in the United States and worldwide. There was no significant temporal trend for "ganglion cyst," "de Quervain's tenosynovitis," "lateral epicondylitis," and "distal radius fracture." There were no significant temporal trend for "scaphoid fracture" in the United States and a decreasing trend worldwide. There was significant seasonal variation in search term interest for "finger fracture" in the United States, "finger fracture" worldwide, and "scaphoid fracture" in the United States, with popularity peaking in the fall.

CONCLUSIONS

Despite growth in global internet usage, internet search interest has remained stagnant for many common hand surgery conditions, which may represent a shifting preference for patients to obtain health information from other resources. Internet search interest for traumatic hand conditions corresponds to seasonal variations in fracture epidemiology and peaks in the fall season.

Evaluation of Online Artificial Intelligence-Generated Information on Common Hand Procedures

PURPOSE

The purpose of this study was to analyze the quality and readability of the information generated by an online artificial intelligence (AI) platform regarding 4 common hand surgeries and to compare AI-generated responses to those provided in the informational articles published by the American Society for Surgery of the Hand (ASSH) HandCare website.

METHODS

An open AI model (ChatGPT) was used to answer questions commonly asked by patients on 4 common hand surgeries (carpal tunnel release, cubital tunnel release, trigger finger release, and distal radius fracture fixation). These answers were evaluated for medical accuracy, quality and readability and compared to answers derived from the ASSH HandCare materials.

RESULTS

For the AI model, the Journal of the American Medical Association benchmark criteria score was 0/4, and the DISCERN score was 58 (considered good). The areas in which the AI model lost points were primarily related to the lack of attribution, reliability and currency of the source material. For AI responses, the mean Flesch Kinkaid Reading Ease score was 15, and the Flesch Kinkaid Grade Level was 34, which is considered to be college level. For comparison, ASSH HandCare materials scored 3/4 on the Journal of the American Medical Association Benchmark, 71 on DISCERN (excellent), 9 on Flesch Kinkaid Grade Level, and 60 on Flesch Kinkaid Reading Ease score (eighth/ninth grade level).

CONCLUSION

An AI language model (ChatGPT) provided generally high-quality answers to frequently asked questions relating to the common hand procedures queried, but it is unclear when or where these answers came from without citations to source material. Furthermore, a high reading level was required to comprehend the information presented. The AI software repeatedly referenced the need to discuss these questions with a surgeon, the importance of shared decision-making and individualized care, and compliance with surgeon treatment recommendations.

CLINICAL RELEVANCE

As novel AI applications become increasingly mainstream, hand surgeons must understand the limitations and ramifications these technologies have for patient care.

Health Information on Pre-Exposure Prophylaxis From Search Engines and Twitter: Readability Analysis

BACKGROUND

Pre-exposure prophylaxis (PrEP) is proven to prevent HIV infection. However, PrEP uptake to date has been limited and inequitable. Analyzing the readability of existing PrEP-related information is important to understand the potential impact of available PrEP information on PrEP uptake and identify opportunities to improve PrEP-related education and communication.

OBJECTIVE

We examined the readability of web-based PrEP information identified using search engines and on Twitter. We investigated the readability of web-based PrEP documents, stratified by how the PrEP document was obtained on the web, information source, document format and communication method, PrEP modality, and intended audience.

METHODS

Web-based PrEP information in English was systematically identified using search engines and the Twitter API. We manually verified and categorized results and described the method used to obtain information, information source, document format and communication method, PrEP modality, and intended audience. Documents were converted to plain text for the analysis and readability of the collected documents was assessed using 4 readability indices. We conducted pairwise comparisons of readability based on how the PrEP document was obtained on the web, information source, document format, communication method, PrEP modality, and intended audience, then adjusted for multiple comparisons.

RESULTS

A total of 463 documents were identified. Overall, the readability of web-based PrEP information was at a higher level (10.2-grade reading level) than what is recommended for health information provided to the general public (ninth-grade reading level, as suggested by the Department of Health and Human Services). Brochures (n=33, 7% of all identified resources) were the only type of PrEP materials that achieved the target of ninth-grade reading level.

CONCLUSIONS

Web-based PrEP information is often written at a complex level for potential and current PrEP users to understand. This may hinder PrEP uptake for some people who would benefit from it. The readability of PrEP-related information found on the web should be improved to align more closely with health communication guidelines for reading level to improve access to this important health information, facilitate informed decisions by those with a need for PrEP, and realize national prevention goals for PrEP uptake and reducing new HIV infections in the United States.

Patient education information material assessment criteria: A scoping review

BACKGROUND

Patient education information material (PEIM) is an essential component of patient education programs in increasing patients' ability to cope with their diseases. Therefore, it is essential to consider the criteria that will be used to prepare and evaluate these resources.

OBJECTIVE

This paper aims to identify these criteria and recognize the tools or methods used to evaluate them.

METHODS

National and international databases and indexing banks, including PubMed, Scopus, Web of Science, ProQuest, the Cochrane Library, Magiran, SID and ISC, were searched for this review. Original or review articles, theses, short surveys, and conference papers published between January 1990 and June 2022 were included.

RESULTS

Overall, 4688 documents were retrieved, of which 298 documents met the inclusion criteria. The criteria were grouped into 24 overarching criteria. The most frequently used criteria were readability, quality, suitability, comprehensibility and understandability.

CONCLUSION

This review has provided empirical evidence to identify criteria, tools, techniques or methods for developing or evaluating a PEIM. The authors suggest that developing a comprehensive tool based on these findings is critical for evaluating the overall efficiency of PEIM using effective criteria.

Readability of Online Foot and Ankle Surgery Patient Education Materials

Background. Online health education resources are frequently accessed by patients seeking information on orthopaedic conditions and procedures. The objectives of this study were to assess the readability of information provided by the American Orthopaedic Foot and Ankle Society (AOFAS) and compare current levels of readability with previous online material. Methods. This study examined 115 articles classified as "Conditions" or "Treatments" on FootCareMD.org. Readability was assessed using the 6 readability assessment tools: Flesch Reading Ease, Flesch-Kincaid Grade Level (FKGL), Gunning Fog Score, Simple Measure of Gobbledygook (SMOG) Index, Coleman-Liau Index, and the Automated Readability Index. Results. The mean readability score across all metrics ranged from 9.1 to 12.1, corresponding to a 9th- to 12th-grade reading level, with a mean FKGL of 9.2 ± SD 1.1 (range: 6.3-15.0). No articles were written below the recommended US sixth-grade reading level, with only 3 articles at or below an eighth-grade level. Treatment articles had higher mean readability grade levels than condition articles (P = .03). Conclusion. Although the volume and quality of the AOFAS resource Web site has increased, readability of information has worsened since 2008 and remains higher than the recommended reading level for optimal comprehension by the general population.Levels of Evidence: Level IV:Retrospective quantitative analysis.

Flesch-Kincaid Measure as Proxy of Socio-Economic Status on Twitter

Social media gives researchers an invaluable opportunity to gain insight into different facets of human life. Researchers put a great emphasis on categorizing the socioeconomic status (SES) of individuals to help predict various findings of interest. Forum uses, hashtags and chatrooms are common tools of conversations grouping. Crowdsourcing involves gathering intelligence to group online user community based on common interest. This paper provides a mechanism to look at writings on social media and group them based on their academic background. We analyzed online forum posts from various geographical regions in the US and characterized the readability scores of users. Specifically, we collected 10,000 tweets from the members of US Senate and computed the Flesch-Kincaid readability score. Comparing the Senators’ tweets to the ones from average internet users, we note 1) US Senators’ readability based on their tweets rate is much higher, and 2) immense difference among average citizen’s score compared to those of US Senators is attributed to the wide spectrum of academic attainment.