Readability of the Most Commonly Accessed Online Patient Education Materials Pertaining to Surgical Treatments of the Spine

Using Large Language Models to Address Health Literacy in mHealth: Case Report

The innate complexity of medical topics often makes it challenging to produce educational content for the public. Although there are resources available to help authors appraise the complexity of their content, there are woefully few resources available to help authors reduce that complexity after it occurs. In this case study, we evaluate using ChatGPT to reduce the complex language used in health-related educational materials. ChatGPT adapted content from the SmartSHOTS mobile application, which is geared toward caregivers of children aged 0 to 24 months. SmartSHOTS helps reduce barriers and improve adherence to vaccination schedules. ChatGPT reduced complex sentence structure and rewrote content to align with a third-grade reading level. Furthermore, using ChatGPT to edit content already written removes the potential for unnoticed, artificial intelligence-produced inaccuracies. As an editorial tool, ChatGPT was effective, efficient, and free to use. This article discusses the potential of ChatGPT as an effective, time-efficient, and open-source method for editing health-related educational materials to reflect a comprehendible reading level.

Readability of Patient-Reported Outcomes in Spine Surgery and Implications for Health Literacy

STUDY DESIGN

Observational study.

OBJECTIVE

To evaluate the readability of commonly utilized patient-reported outcome measures (PROMs) in spine surgery.

SUMMARY OF BACKGROUND DATA

Although studies have evaluated patient education materials, discharge instructions, and informed consent forms in spine surgery, there is a dearth of literature on the readability of PROMs despite widespread health illiteracy. Without knowledge of PROM readability, it is unclear whether these measures are able to be understood by the average spine patient.

MATERIALS AND METHODS

We analyzed all commonly utilized nonvisual PROMs within the spinal literature and uploaded PROMs into an online readability calculator. The Flesch Reading Ease Score (FRES) and Simple Measure of Gobbledygook (SMOG) Index were collected. A FRES>79 or SMOG<7 was considered readable by the general population per American Medical Association and Centers for Disease Control guidelines. A stricter threshold recommended in health care (SMOG <6 or FRES>89) was then used to further review readability.

RESULTS

Seventy-seven PROMs were included. Based on FRES, the mean readability of all PROMs was 69.2 ± 17.2 (range, 10-96.4), indicating an average eighth to ninth-grade reading level. The mean readability score categorized by the SMOG Index was 8.12 ± 2.65 (range, 3.1-25.6), representing an eighth-grade reading level. Compared with the reading level of the general population, 49 (63.6%) PROMs are written above the United States literacy level, according to FRES. Using stricter definitions of readability, 8 PROMs were considered readable, including PROM Information System pain behavior (FRES: 96.4 and SMOG: 5.2), PROM Information System sleep disturbance (SMOG: 5.6), Neck Pain and Disability Scale (SMOG: 4.3), and Zung Depression Scale (SMOG: 3.1).

CONCLUSIONS

Most PROMs utilized in spine surgery require an average reading competency far above the average patient's comprehension. This may have a meaningful impact on understanding PROM instruments and may affect the accuracy of complete surveys and the rates of incompletion.

Examination of the reliability and readability of Chatbot Generative Pretrained Transformer's (ChatGPT) responses to questions about orthodontics and the evolution of these responses in an updated version

INTRODUCTION

This study aimed to assess the reliability and readability of Chatbot Generative Pretrained Transformer (ChatGPT) responses to questions about orthodontics and the evolution of these responses in an updated version.

METHODS

Frequently asked questions about orthodontics by laypeople on Web sites were determined using the Google Search Tool. These questions were asked to both ChatGPT's March 23 version and May 24 version on April 20, 2023, and July 12, 2023, respectively. Responses were assessed for readability and reliability using the Flesch-Kincaid and DISCERN tests.

RESULTS

The mean DISCERN value for general questions was 2.96 ± 0.05, 3.04 ± 0.06, 2.38 ± 0.27, and 2.82 ± 0.31 for treatment-related questions; the mean Flesch-Kincaid Reading Ease score for general questions was 29.28 ± 8.22, 25.12 ± 7.39, 47.67 ± 10.77, and 41.60 ± 9.54 for treatment-related questions; mean Flesch-Kincaid Grade Level for general questions was 14.52 ± 1.48 and 14.04 ± 1.25 and 11.90 ± 2.08 and 11.41 ± 1.88 for treatment-related questions; in first and second evaluations respectively (P = 0.001).

CONCLUSIONS

In the second evaluation, the reliability of the answers given to general questions and treatment-related questions increased. However, in both evaluations, the reliability of the answers was found to be moderate according to the DISCERN tool. On the second evaluation, Flesch Reading Ease Scores for both general questions and treatment-related questions decreased, meaning that the readability of the new response texts became more difficult. Flesch-Kincaid Grade Level results were found at the college graduate level in the first and second evaluations for general questions and at the high school level in the first and second evaluations for treatment-related questions.

The Impact of Socioeconomic Status on the Presence of Advance Care Planning Documents in Patients With Acute Cervical Spinal Cord Injury

INTRODUCTION

Patients presenting with spinal cord injury (SCI) often times have notable deficits or polytrauma and may require urgent decision making for early management. However, their presentation may affect decision-making ability. Although advance care planning (ACP) may help guide spine surgeons as to patient preferences, the rate at which they are available and disparities in ACP completion are still not understood. The objective of this study was to evaluate disparities in the completion of ACP among patients with acute SCI.

METHODS

All patients presenting with cervical SCI to the emergency department at an urban, tertiary level I trauma center from 2010 to 2021 were identified from a prospective database of all consults evaluated by the spine service. Each patient's medical record was reviewed to assess for the presence of ACP documents such as living will, power of attorney, or advance directive. Community-level socioeconomic status was assessed using the Distressed Communities Index. Bivariable and multivariable analyses were performed.

RESULTS

We identified 424 patients: 104 (24.5%) of whom had ACP. Patients with ACP were older (64.8 versus 56.5 years, P = 0.001), more likely White (78.8% versus 71.9%, P = 0.057), and present with ASIA Impairment Scale grade A SCI (21.2% versus 12.8%, P = 0.054), although the latter two did not reach statistical significance. On multivariable logistic regression, patients residing in at-risk communities were significantly less likely to have ACP documents compared with those in prosperous communities (odds ratio [OR]: 0.29, P = 0.03). Although patients living in distressed communities were less likely to complete ACP compared with those in prosperous communities (OR 0.50, P = 0.066), this did not meet statistical significance. Female patients were also less likely to have ACP (OR: 0.43, P = 0.005).

CONCLUSION

Female patients and those from at-risk communities are markedly less likely to complete ACP. Attention to possible disparities during admission and ACP discussions may help ensure that patients of all backgrounds have treatment goals documented.

Readability of online patient education materials for shoulder instability surgery in English and Spanish

BACKGROUND

Online patient education materials (OPEMs) exist to inform patient medical decisions, yet the average adult in the United States reads at an eighth-grade level and 50% of Medicaid patients read at or below a fifth-grade level. To appropriately meet US health literacy needs, the American Medical Association and National Institutes of Health recommend that patient education materials not exceed a sixth-grade level. The purpose of this study was to assess and compare the readability of English and Spanish online patient education materials pertaining to shoulder instability surgery.

METHODS

Google searches of the terms "shoulder instability surgery" and "cirugía de inestabilidad de hombro'' were conducted to include 25 eligible online patient education materials OPEMs per language. English OPEM readability was calculated using Flesch-Kincaid Grade Level, Flesch Reading Ease, Flesch Reading Ease Grade Level, Gunning-Fog Index, Coleman-Liau Index, and Simple Measure of Gobbledygook. Spanish OPEM readability was assessed using Fernandez-Huerta Index (FHI) (the Spanish equivalent of Flesch Reading Ease), FHI Grade Level, Gutiérrez de Polini's Fórmula de comprensibilidad, and INFLESZ.

RESULTS

Readability index analysis revealed that the mean Flesch Reading Ease of English online patient education materials was significantly lower than the mean FHI of Spanish online patient education materials. English materials were also found to be written at a significantly higher grade level than Spanish materials.

CONCLUSIONS

Shoulder instability surgery online patient education materials in both English and Spanish are written at higher reading levels than recommended by the American Medical Association and National Institutes of Health, though Spanish online patient education materials were more readable on average.

Digital Health for Patients Undergoing Spine Surgery: A Systematic Review

OBJECTIVE

Digital health tools, including smartphone applications (apps), websites, and online search engines, are increasingly being utilized for health data collection and patient education. Studies have shown that these tools can help disseminate information widely and even help guide patients through acute surgical episodes. We aimed to search the literature to summarize available studies on using digital health tools for patients undergoing spine surgery.

METHODS

We conducted a systematic review of PubMed MEDLINE, Elsevier EMBASE, and Elsevier Scopus databases, as well as ClinicalTrials.gov up to March 11, 2022.

RESULTS

Forty-four full-text articles were included and qualitatively analyzed. Studies were broadly grouped into those that analyzed the quality of web-based materials for patients, the quality of YouTube videos for spine surgery, the development, feasibility, and implementation of mobile apps for patients, and randomized controlled trials for integrating mobile apps into perioperative care.

CONCLUSIONS

We presented a systematic review analyzing the current landscape of digital health for patients undergoing spine surgery. Internet patient education materials in searchable websites and YouTube videos are of poor quality, lacking in readability to the average patient and robustness of information needed for patients to make informed decisions about pursuing spine surgery. However, there lies promise in digital apps developed to guide patients through surgery and collect postoperative outcomes.

Readability of spine-related patient education materials: a standard method for improvement

PURPOSE

Orthopaedic patient education materials (PEMs) have repeatedly been shown to be well above the recommended reading level by the National Institute of Health and American Medical Association. The purpose of this study is to create a standardized method to improve the readability of PEMs describing spine-related conditions and injuries. It is hypothesized that reducing the usage of complex words (≥ 3 syllables) and reducing sentence length to < 15 words per sentence improves readability of PEMs as measured by all seven readability formulas used.

METHODS

OrthoInfo.org was queried for spine-related PEMs. The objective readability of PEMs was evaluated using seven unique readability formulas before and after applying a standardized method to improve readability while preserving critical content. This method involved reducing the use of > 3 syllable words and ensuring sentence length is < 15 words. Paired samples t-tests were conducted to assess relationships with the cut-off for statistical significance set at p < 0.05.

RESULTS

A total of 20 spine-related PEM articles were used in this study. When comparing original PEMs to edited PEMs, significant differences were seen among all seven readability scores and all six numerical descriptive statistics used. Per the Flesch Kincaid Grade level readability formula, one original PEM (5%) versus 15 edited PEMs (75%) met recommendations of a sixth-grade reading level.

CONCLUSION

The current study shows that using this standardized method significantly improves the readability of spine-related PEMs and significantly increased the likelihood that PEMs will meet recommendations for being at or below the sixth-grade reading level.

TLIF Online Videos for Patient Education-Evaluation of Comprehensiveness, Quality, and Reliability

Over the last few decades, the number of lumbar interbody fusion surgeries performed has been constantly increasing, with transforaminal lumbar interbody fusion (TLIF) being one of the most common surgical techniques. Due to easy accessibility, patients frequently use YouTube to obtain information on health-related issues. Consequently, online video platforms may be a valuable tool for patient education. The aim of this study was to assess the quality, reliability, and comprehensiveness of online videos on TLIF. We screened 180 videos on YouTube, yielding a total of 30 videos that met the inclusion criteria. These videos were evaluated using Global Quality Scale, DISCERN reliability tool, and JAMA Benchmark Score, and assessed in regard to their comprehensiveness and coverage of relevant aspects. At the time of rating, the videos had between 9188 and 1,530,408 views and between 0 and 3344 likes. The median rater assessment for all videos was "moderate quality". GQS and subjective grades showed a moderate to strong statistically significant association with views and likes. Considering this association of GQS and subjective grade with views and likes, these criteria could be used by laypersons to identify good-quality content. Nevertheless, there is an urgent need for peer-reviewed content that covers all of the relevant aspects.

Dr Google - assessing the reliability and readability of information on general surgical procedures found via search engines

BACKGROUND

The most common general surgical emergency operations are laparoscopic appendicectomy, laparoscopic cholecystectomy, hernia repair, hemorrhoidectomy and colectomy. Patients commonly perform an internet search for more information prior to undergoing surgery, which can lead to an inappropriate understanding of their procedure. The aim is to assess the quality of information available on three of the most used search engines.

METHODS

A search was conducted on Google.com, Bing.com and Yahoo.com using the terms related to laparoscopic appendicectomy, laparoscopic cholecystectomy, hemorrhoidectomy, hernia repair and colectomy. First 20 results from each search engine were collected for evaluation. Results were excluded if they were sponsored, duplicates, academic publications, advertisements, forums, audiovisual tools, social media or any non-English information. Included results were assessed for reliability using DISCERN and JAMA benchmark score. Readability was assessed using Flesch Reading Ease (FRE) Score and Simple Measure of Gobbledygook (SMOG).

RESULTS

Hundred and ninety-seven websites were analysed, 44.7% were published by institutions, 34.5% by health websites and 20.8% by independent surgeons. Mean DISCERN scores for Institutions was 54.6 ± 11.3, independent surgeons 45.9 ± 11.4 and health websites 58.7 ± 10.3. Mean JAMA score for Institutions was 1.0 ± 1.0, independent surgeons 0.1 ± 0.4 and health websites 1.7 ± 1.1. FRE scores for institutions was 51.6 ± 10.3, independent surgeons 40.9 ± 10.2, and health websites 45.7 ± 12.3. SMOG scores were 9.8 ± 1.5 for institutions, 11.4 ± 1.6 for independent surgeons and 10.6 ± 1.7 for health websites.

CONCLUSION

Health information on common general surgical procedures found on search engines are generally fair to good quality but still above the suggested reading level of the population. Information on surgical procedures should be written at recommended reading level of 13-14 years old.

Readability of Online Spine Patient Education Resources

OBJECTIVE

We assessed the readability of spine-related patient education materials on professional society websites to determine whether this had improved since last studied. We also compared the readability of these materials to a more patient-centered source, such as WebMD.

METHODS

Patient education pages from the American Association of Neurologic Surgeons (AANS), North American Spine Society (NASS), and spine-related pages from the American Academy of Orthopaedic Surgeons (AAOS), and WebMD were reviewed. Readability was evaluated using the Flesch Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE) formulas. The mean FKGL and FRE scores of the societies were compared using one-way analysis of variance. The rate of a reading level at or below an eighth grade level was compared using the χ² test.

RESULTS

We analyzed a total of 156 sites. The mean FKGL score for the professional society sites was 11.4. The mean FRE score for the professional societies was 45.8, with 14.4% written at or below an eighth grade reading level. We found a significant difference in the FKGL scores and materials at or below the eighth grade level between the AAOS and AANS and AAOS and NASS. The mean FKGL and FRE scores for WebMD were 7.57 and 68.1, respectively, with a significant difference compared with the scores for the AAOS, NASS, and AANS. In addition, 80% of the WebMD materials had been written at or below the eighth grade reading level. A significant difference compared with the AANS and NASS (P < 0.0001) but not for the AAOS (P = 0.059).

CONCLUSIONS

The average readability of spine-related topics exceeded the eighth grade reading level. The AAOS resources had better readability compared with the NASS and AANS. We found no improvement in readability since last studied. The readability of professional societies' materials was significantly worse than those from WebMD.