Multiple Automated Health Literacy Assessments of Written Health Information: Development of the SHeLL (Sydney Health Literacy Lab) Health Literacy Editor v1

Women's interest, knowledge, and attitudes relating to anti-Mullerian hormone testing: a randomized controlled trial

STUDY QUESTION

What is the impact of co-designed, evidence-based information regarding the anti-Mullerian hormone (AMH) test on women's interest in having the test?

SUMMARY ANSWER

Women who viewed the evidence-based information about the AMH test had lower interest in having an AMH test than women who viewed information produced by an online company selling the test direct-to-consumers.

WHAT IS KNOWN ALREADY

Online information about AMH testing often has unfounded claims about its ability to predict fertility and conception, and evidence suggests that women seek out and are recommended the AMH test as a measure of their fertility potential.

STUDY DESIGN, SIZE, DURATION

An online randomized trial was conducted from November to December 2022. Women were randomized (double-blind, equal allocation) to view one of two types of information: co-designed, evidence-based information about the AMH test (intervention), or existing information about the AMH test from a website which markets the test direct-to-consumers (control). A total of 967 women were included in the final analysis.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Participants were women recruited through an online panel, who were aged 25-40 years, living in Australia or The Netherlands, had never given birth, were not currently pregnant but would like to have a child now or in the future, and had never had an AMH test. The primary outcome was interest in having an AMH test (seven-point scale; 1 = definitely NOT interested to 7 = definitely interested). Secondary outcomes included attitudes, knowledge, and psychosocial and behavioural outcomes relating to AMH testing.

MAIN RESULTS AND THE ROLE OF CHANCE

Women who viewed the evidence-based information about the AMH test had lower interest in having an AMH test (MD = 1.05, 95% CI = 0.83-1.30), less positive attitudes towards (MD = 1.29, 95% CI = 4.57-5.70), and higher knowledge about the test than women who viewed the control information (MD = 0.75, 95% CI = 0.71-0.82).

LIMITATIONS, REASONS FOR CAUTION

The sample was more highly educated than the broader Australian and Dutch populations and some measures (e.g. influence on family planning) were hypothetical in nature.

WIDER IMPLICATIONS OF THE FINDINGS

Women have higher knowledge of and lower interest in having the AMH test when given evidence-based information about the test and its limitations. Despite previous studies suggesting women are enthusiastic about AMH testing to learn about their fertility potential, we demonstrate that this enthusiasm does not hold when they are informed about the test's limitations.

STUDY FUNDING/COMPETING INTEREST(S)

This project was supported by an NHMRC Emerging Leader Research Fellowship (2009419) and the Australian Health Research Alliance's Women's Health Research, Translation and Impact Network EMCR award. B.W.M. reports consultancy for ObsEva and Merck and travel support from Merck. D.L. is the Medical Director of, and holds stock in, City Fertility NSW and reports consultancy for Organon and honoraria from Ferring, Besins, and Merck. K.H. reports consultancy and travel support from Merck and Organon. K.M. is a director of Health Literacy Solutions that owns a licence of the Sydney Health Literacy Lab Health Literacy Editor. No other relevant disclosures exist.

TRIAL REGISTRATION NUMBER

ACTRN12622001136796.

TRIAL REGISTRATION DATE

17 August 2022.

DATE OF FIRST PATIENT’S ENROLMENT

21 November 2022.

A Decision Aid for Patients Considering Surgery for Sciatica: Codesign and User-Testing With Patients and Clinicians

BACKGROUND

Surgery can help patients with leg pain caused by sciatica recover faster, but by 12 months outcomes are similar to nonsurgical management. For many the decision to have surgery may require reflection, and patient decision aids are an evidence-based clinical tool that can help guide patients through this decision.

OBJECTIVE

The aim of this study was to develop and refine a decision aid for patients with sciatica who are deciding whether to have surgery or 'wait and see' (i.e., try nonsurgical management first).

DESIGN

Semistructured interviews with think-aloud user-testing protocol.

PARTICIPANTS

Twenty clinicians and 20 patients with lived experience of low back pain or sciatica.

OUTCOME MEASURES

Items from Technology Acceptance Model, Preparation for Decision Making Scale and Decision Quality Instrument for Herniated Disc 2.0 (knowledge instrument).

METHODS

The prototype integrated relevant research with working group perspectives, decision aid standards and health literacy guidelines. The research team refined the prototype through seven rounds of user-testing, which involved discussing user-testing feedback and implementing changes before progressing to the next round.

RESULTS

As a result of working group feedback, the decision aid was divided into sections: before, during and after a visit to the surgeon. Across all rounds of user-testing, clinicians rated the resource 5.9/7 (SD = 1.0) for perceived usefulness, and 6.0/7 for perceived ease of use (SD = 0.8). Patients reported the decision aid was easy to understand, on average correctly answering 3.4/5 knowledge questions (SD = 1.2) about surgery for sciatica. The grade reading score for the website was 9.0. Patients scored highly on preparation for decision-making (4.4/5, SD = 0.7), suggesting strong potential to empower patients. Interview feedback showed that patients and clinicians felt the decision aid would encourage question-asking and help patients reflect on personal values.

CONCLUSIONS

Clinicians found the decision aid acceptable, patients found it was easy to understand and both groups felt it would empower patients to actively engage in their care and come to an informed decision that aligned with personal values. Input from the working group and user-testing was crucial for ensuring that the decision aid met patient and clinician needs.

PATIENT OR PUBLIC CONTRIBUTION

Patients and clinicians contributed to prototype development via the working group.

Communicating the Imperfect Diagnostic Accuracy of COVID-19 Rapid Antigen Self-Tests: An Online Randomized Experiment

OBJECTIVE

To investigate the potential impacts of optimizing coronavirus disease 2019 (COVID-19) rapid antigen test (RAT) self-testing diagnostic accuracy information.

DESIGN

Online randomized experiment using hypothetical scenarios: in scenarios 1 to 3 (RAT result positive), the posttest probability was considered to be very high (likely true positives), and in scenarios 4 and 5 (RAT result negative), the posttest probability was considered to be moderately high (likely false negatives).

SETTING

December 12 to 22, 2022, during the mixed-variant Omicron wave in Australia.

PARTICIPANTS

Australian adults. Intervention: diagnostic accuracy of a COVID-19 self-RAT presented in a health literacy-sensitive way; usual care: diagnostic accuracy information provided by the manufacturer; control: no diagnostic accuracy information.

MAIN OUTCOME MEASURE

Intention to self-isolate.

RESULTS

A total of 226 participants were randomized (control n = 75, usual care n = 76, intervention n = 75). More participants in the intervention group correctly interpreted the meaning of the diagnostic accuracy information (P = 0.08 for understanding sensitivity, P < 0.001 for understanding specificity). The proportion who would self-isolate was similar across scenarios 1 to 3 (likely true positives). The proportion was higher in the intervention group than in the control for scenarios 4 and 5 (likely false negatives). These differences were not statistically significant. The largest potential effect was seen in scenario 5 (dinner party with confirmed cases, the person has symptoms, negative self-RAT result), with 63% of the intervention group and 49% of the control group indicating they would self-isolate (absolute difference 13.3%, 95% confidence interval: -2% to 30%, P = 0.10).

CONCLUSION

Health literacy sensitive formatting supported participant understanding and recall of diagnostic accuracy information. This may increase community intentions to self-isolate when there is a likely false-negative self-RAT result. Trial registration: Australia New Zealand Clinical Trial Registry (ACTRN12622001517763).

HIGHLIGHTS

Community-based diagnostic accuracy studies of COVID-19 self-RATs indicate substantially lower sensitivity (and higher risk of false-negative results) than the manufacturer-supplied information on most government public Web sites.This online randomized study found that a health literacy-sensitive presentation of the imperfect diagnostic accuracy COVID-19 self-RATs supported participant understanding and recall of diagnostic accuracy information.Health literacy-sensitive presentation may increase community intentions to self-isolate after a negative test result where the posttest probability is still moderately high (i.e., likely false-negative result).To prevent the onward spread of infection, efforts to improve communication about the high risk of false-negative results from COVID-19 self-RATs are urgently needed.

Assessing the Readability of Clinical Trial Consent Forms for Surgical Specialties

INTRODUCTION

To evaluate the readability of surgical clinical trial consent forms and compare readability across surgical specialties.

METHODS

We conducted a cross-sectional analysis of surgical clinical trial consent forms available on ClinicalTrials.gov to quantitatively evaluate readability, word count, and length variations among different specialties. The analysis was performed between November 2022 and January 2023. A total of 386 surgical clinical trial consent forms across 14 surgical specialties were included.

RESULTS

The main outcomes were language complexity (measured using Flesch-Kincaid Grade Level), number of words (measured as word count), time to read (measured at reading speeds of 240 per min), and readability (measured by Flesch Reading Ease Score, Gunning Frog Index, Simple Measures of Gobbledygook Index, FORCAST, and Automated Readability Index). The surgical consent forms were a mean (standard deviation) of 2626 (1668) words long, with a mean of 12:53 min to read at 240 words per min. None of the surgical specialties had an average readability level of sixth grade or lower across all six indices, and only 16 out of 386 (4%) clinical trials met the recommended reading level. Furthermore, there was no significant difference in reading grade level between surgical specialties based on the Flesch-Kincaid Grade Level and Flesch Reading Ease indices.

CONCLUSIONS

Our findings suggest that current surgical clinical trial consent documents are too long and complex, exceeding the recommended sixth-grade reading level. Ensuring readable clinical trial consent forms is not only ethically responsible but also crucial for protecting patients' rights and well-being by facilitating informed decision-making.

New Frontiers in Health Literacy: Using ChatGPT to Simplify Health Information for People in the Community

BACKGROUND

Most health information does not meet the health literacy needs of our communities. Writing health information in plain language is time-consuming but the release of tools like ChatGPT may make it easier to produce reliable plain language health information.

OBJECTIVE

To investigate the capacity for ChatGPT to produce plain language versions of health texts.

DESIGN

Observational study of 26 health texts from reputable websites.

METHODS

ChatGPT was prompted to 'rewrite the text for people with low literacy'. Researchers captured three revised versions of each original text.

MAIN MEASURES

Objective health literacy assessment, including Simple Measure of Gobbledygook (SMOG), proportion of the text that contains complex language (%), number of instances of passive voice and subjective ratings of key messages retained (%).

KEY RESULTS

On average, original texts were written at grade 12.8 (SD = 2.2) and revised to grade 11.0 (SD = 1.2), p < 0.001. Original texts were on average 22.8% complex (SD = 7.5%) compared to 14.4% (SD = 5.6%) in revised texts, p < 0.001. Original texts had on average 4.7 instances (SD = 3.2) of passive text compared to 1.7 (SD = 1.2) in revised texts, p < 0.001. On average 80% of key messages were retained (SD = 15.0). The more complex original texts showed more improvements than less complex original texts. For example, when original texts were ≥ grade 13, revised versions improved by an average 3.3 grades (SD = 2.2), p < 0.001. Simpler original texts (< grade 11) improved by an average 0.5 grades (SD = 1.4), p < 0.001.

CONCLUSIONS

This study used multiple objective assessments of health literacy to demonstrate that ChatGPT can simplify health information while retaining most key messages. However, the revised texts typically did not meet health literacy targets for grade reading score, and improvements were marginal for texts that were already relatively simple.

The future of allergic rhinitis management: A partnership between healthcare professionals and patients

Allergic rhinitis (AR) is a chronic respiratory condition that internationally continues to be burdensome and impacts quality of life. Despite availability of medicines and guidelines for healthcare providers for the optimal management of AR, optimisation of its management in the community continues to be elusive. The reasons for this are multi-faceted and include both environmental and healthcare related factors. One factor that we can no longer ignore is that AR management is no longer limited to the domain of healthcare provider and that people with AR make their own choices when choosing how to manage their condition, without seeking advice from a health care provider. We must build a bridge between healthcare provider knowledge and guidelines and patient decision-making. With this commentary, we propose that a shared decision-making approach between healthcare professionals and people with AR be developed and promoted, with a focus on patient health literacy. As custodians of AR knowledge, we have a responsibility to ensure it is accessible to those that matter most-the people with AR.

Helping patient educators meet health literacy needs: End-user testing and iterative development of an innovative health literacy editing tool

Objective

The Sydney Health Literacy Lab (SHeLL) Editor is an online text-editing tool that provides real-time assessment and feedback on written health information (assesses grade reading score, complex language, passive voice). This study aimed to explore how the design could be further enhanced to help health information providers interpret and act on automated feedback.

Methods

The prototype was iteratively refined across four rounds of user-testing with health services staff (N = 20). Participants took part in online interviews and a brief follow-up survey using validated usability scales (System Usability Scale, Technology Acceptance Model). After each round, Yardley's (2021) optimisation criteria guided which changes would be implemented.

Results

Participants rated the Editor as having adequate usability (M = 82.8 out of 100, SD = 13.5). Most modifications sought to reduce information overload (e.g. simplifying instructions for new users) or make feedback motivating and actionable (e.g. using frequent incremental feedback to highlight changes to the text altered assessment scores).

Conclusion

terative user-testing was critical to balancing academic values and the practical needs of the Editor's target users. The final version emphasises actionable real-time feedback and not just assessment.

Innovation

The Editor is a new tool that will help health information providers apply health literacy principles to written text.

Can we use ChatGPT for Mental Health and Substance Use Education? Examining Its Quality and Potential Harms

BACKGROUND

The use of generative artificial intelligence, more specifically large language models (LLMs), is proliferating, and as such, it is vital to consider both the value and potential harms of its use in medical education. Their efficiency in a variety of writing styles makes LLMs, such as ChatGPT, attractive for tailoring educational materials. However, this technology can feature biases and misinformation, which can be particularly harmful in medical education settings, such as mental health and substance use education. This viewpoint investigates if ChatGPT is sufficient for 2 common health education functions in the field of mental health and substance use: (1) answering users' direct queries and (2) aiding in the development of quality consumer educational health materials.

OBJECTIVE

This viewpoint includes a case study to provide insight into the accessibility, biases, and quality of ChatGPT's query responses and educational health materials. We aim to provide guidance for the general public and health educators wishing to utilize LLMs.

METHODS

We collected real world queries from 2 large-scale mental health and substance use portals and engineered a variety of prompts to use on GPT-4 Pro with the Bing BETA internet browsing plug-in. The outputs were evaluated with tools from the Sydney Health Literacy Lab to determine the accessibility, the adherence to Mindframe communication guidelines to identify biases, and author assessments on quality, including tailoring to audiences, duty of care disclaimers, and evidence-based internet references.

RESULTS

GPT-4's outputs had good face validity, but upon detailed analysis were substandard in comparison to expert-developed materials. Without engineered prompting, the reading level, adherence to communication guidelines, and use of evidence-based websites were poor. Therefore, all outputs still required cautious human editing and oversight.

CONCLUSIONS

GPT-4 is currently not reliable enough for direct-consumer queries, but educators and researchers can use it for creating educational materials with caution. Materials created with LLMs should disclose the use of generative artificial intelligence and be evaluated on their efficacy with the target audience.

Integrating consumer perspectives into a large-scale health literacy audit of health information materials: learnings and next steps

BACKGROUND

Health information is less effective when it does not meet the health literacy needs of its consumers. For health organisations, assessing the appropriateness of their existing health information resources is a key step to addressing this issue. This study describes novel methods for a consumer-centred large-scale health literacy audit of existing resources and reflects on opportunities to further refine the method.

METHODS

This audit focused on resources developed by NPS MedicineWise, an Australian not-for-profit that promotes safe and informed use of medicines. The audit comprised 4 stages, with consumers engaged at each stage: 1) Select a sample of resources for assessment; 2) Assess the sample using subjective (Patient Education Materials Assessment Tool) and objective (Sydney Health Literacy Lab Health Literacy Editor) assessment tools; 3) Review audit findings through workshops and identify priority areas for future work; 4) Reflect and gather feedback on the audit process via interviews.

RESULTS

Of 147 resources, consumers selected 49 for detailed assessment that covered a range of health topics, health literacy skills, and formats, and which had varied web usage. Overall, 42 resources (85.7%) were assessed as easy to understand, but only 26 (53.1%) as easy to act on. A typical text was written at a grade 12 reading level and used the passive voice 6 times. About one in five words in a typical text were considered complex (19%). Workshops identified three key areas for action: make resources easier to understand and act on; consider the readers' context, needs, and skills; and improve inclusiveness and representation. Interviews with workshop attendees highlighted that audit methods could be further improved by setting clear expectations about the project rationale, objectives, and consumer roles; providing consumers with a simpler subjective health literacy assessment tool, and addressing issues related to diverse representation.

CONCLUSIONS

This audit yielded valuable consumer-centred priorities for improving organisational health literacy with regards to updating a large existing database of health information resources. We also identified important opportunities to further refine the process. Study findings provide valuable practical insights that can inform organisational health actions for the upcoming Australian National Health Literacy Strategy.