The differential effects of presenting uncertainty around benefits and harms on treatment decision making

Laypersons' understanding of statistical concepts commonly used in prescription drug promotion: A review of the research literature

BACKGROUND

The prevalence of direct-to-consumer (DTC) advertising for prescription drugs has led to concerns about how consumers interpret the medical information conveyed in these ads. One strategy for improving lay understanding of medical information involves incorporating quantitative information about a treatment's potential benefits and risks.

OBJECTIVE

This literature review investigates laypersons' interpretations of statistical concepts, expanding on past reviews and including terms that may be used in DTC prescription drug advertising.

METHODS

We searched six databases for articles published from January 2000 to October 2021. Articles were included if they were in English and examined general or lay audiences' comprehension of quantitative or statistical concepts, without limiting the context of the studies to medical situations.

RESULTS

We identified 25 eligible articles. The evidence suggests that likelihood ratios, odds ratios, probabilities, numbers needed to treat/harm, and confidence intervals hinder comprehension of quantitative information. The results are mixed for information presented as frequencies, percentages, absolute risk reduction, and relative risk reduction. The mixed findings could be due to numeracy, framing as risks or benefits, and operationalization of the outcomes. We found no studies examining interpretations of minimum, maximum, central tendency, power, statistical significance, or hazard ratio.

CONCLUSION

Studies spanning several decades have examined how laypeople interpret statistical concepts. While a few terms are consistently studied, many questions still remain on how to make risk information more understandable to lay audiences, particularly those with low numeracy.

Probability and informed consent

In this paper, we illustrate some serious difficulties involved in conveying information about uncertain risks and securing informed consent for risky interventions in a clinical setting. We argue that in order to secure informed consent for a medical intervention, physicians often need to do more than report a bare, numerical probability value. When probabilities are given, securing informed consent generally requires communicating how probability expressions are to be interpreted and communicating something about the quality and quantity of the evidence for the probabilities reported. Patients may also require guidance on how probability claims may or may not be relevant to their decisions, and physicians should be ready to help patients understand these issues.

Communicating Uncertainty in Written Consumer Health Information to the Public: Parallel-Group, Web-Based Randomized Controlled Trial

Background

Uncertainty is integral to evidence-informed decision making and is of particular importance for preference-sensitive decisions. Communicating uncertainty to patients and the public has long been identified as a goal in the informed and shared decision-making movement. Despite this, there is little quantitative research on how uncertainty in health information is perceived by readers.

Objective

The aim of this study was to examine the impact of different uncertainty descriptions regarding the evidence for a treatment effect in a written research summary for the public.

Methods

We developed 8 versions of a research summary on a fictitious drug for tinnitus with varying degrees (Q1), sources (Q2), and magnitudes of uncertainty (Q3). We recruited 2099 members of the German public from a web-based research panel. Of these, 1727 fulfilled the inclusion criteria and were randomly presented with one of these research summaries. Randomization was conducted by using a centralized computer with a random number generator. Web-based recruitment and data collection were fully automated. Participants were not aware of the purpose of the study and alternative presentations. We measured the following outcomes: perception of the treatment effectiveness (primary), certainty in the judgement of treatment effectiveness, perception of the body of evidence, text quality, and intended decision. The outcomes were self-assessed.

Results

For the primary outcome, we did not find a global effect for Q1 and Q2 (P=.25 and P=.73), but we found a global effect for Q3 (P=.048). Pairwise comparisons showed a weaker perception of treatment effectiveness for the research summary with 3 sources of uncertainty compared to the version with 2 sources of uncertainty (P=.04). Specifically, the proportion of the participants in the group with 3 sources of uncertainty that perceived the drug as possibly beneficial was 9% lower than that of the participants in the group with 2 sources of uncertainty (92/195, 47.2% vs 111/197, 56.3%, respectively). The proportion of the participants in the group with 3 sources of uncertainty that considered the drug to be of unclear benefit was 8% higher than that of the participants in the group with 2 sources of uncertainty (72/195, 36.9% vs 57/197, 28.9%, respectively). However, there was no significant difference compared to the version with 1 source of uncertainty (P=.31). We did not find any meaningful differences between the research summaries for the secondary outcomes.

Conclusions

Communicating even a large magnitude of uncertainty for a treatment effect had little impact on the perceived effectiveness. Efforts to improve public understanding of research are needed to improve the understanding of evidence-based health information.

Trial Registration

German Clinical Trials Register DRKS00015911, https://www.drks.de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00015911

International Registered Report Identifier (IRRID)

RR2-10.2196/13425

A review of the effects of uncertainty in public science communication

Uncertainty is inherent to science and science communication. However, the evidence appears mixed regarding whether portraying uncertainty in science communication has positive or negative effects. We review a diverse range of experimental literature (k = 48; from 40 searches and 8000 retrievals), summarize the extant findings, and observe how the effects vary across four different types of communicated uncertainty (deficient, technical, scientific, and consensus uncertainty). The results indicate that most findings of negative effects (such as reduced credibility and beliefs) are from experiments that operationalized uncertainty as disagreement or conflict in science (consensus uncertainty). In this review, consensus uncertainty was never found to have positive effects. In contrast, uncertainty in the form of quantified error ranges and probabilities (technical uncertainty) in these studies has had only positive or null effects, not negative effects. We also highlight frequent moderators of the effects of uncertainty, such as prior beliefs and worldviews.

How is cervical cancer screening information communicated in UK websites? Cross-sectional analysis of content and quantitative presentation formats

OBJECTIVES

To investigate whether UK websites about cervical cancer screening targeted to the public include (1) information about benefits and risks of screening, possible screening results and cervical cancer statistics, (2) quantitative presentation formats recommended in the risk communication literature and (3) appeals for participation and/or informed decision-making.

DESIGN

Cross-sectional analysis of websites using a comprehensive checklist of information items on screening benefits, risks, possible results and cervical cancer statistics.

OUTCOME MEASURES

We recorded the number of websites that contained each of the information items, and the presentation format used for probabilistic information (no quantification provided, verbal quantifiers only, different types of numerical formats and/or graphs). We also recorded the number of websites containing appeals for participation and/or informed decision-making.

SETTING

Websites were identified through the most common Google search terms used in the UK to find information on cervical screening, according to GoogleTrends and a commercial internet-monitoring programme. Two additional websites were identified by the authors as relevant.

RESULTS

After applying exclusion criteria, 14 websites were evaluated, including websites of public and private health service providers, charities, a medical society and a pharmacy. The websites mentioned different benefits, risks of screening and possible results. However, specific content varied between websites. Probabilistic information was often presented using non-recommended formats, including relative risk reductions to express screening benefits, and verbal quantifiers without numbers to express risks. Appeals for participation were present in most websites, with almost half also mentioning informed decision-making.

CONCLUSIONS

UK websites about cervical cancer screening were generally balanced. However, benefits and risks were presented using different formats, potentially hindering comparisons. Additionally, recommendations from the literature to facilitate understanding of quantitative information and facilitate informed decisions were often not followed. Designing websites that adhere to existing recommendations may support informed screening uptake.

Communicating Uncertainty From Limitations in Quality of Evidence to the Public in Written Health Information: Protocol for a Web-Based Randomized Controlled Trial

BACKGROUND

Uncertainty is integral to evidence-informed decision making and is of particular importance for preference-sensitive decisions. Communicating uncertainty to patients and the public has long been identified as a goal in the informed and shared decision-making movement. Despite this, there is little quantitative research on how uncertainty in health information is perceived by readers.

OBJECTIVE

The objective of this study is to design an experiment to examine how different degrees of uncertainty (Q1) and different types of uncertainty (Q2) impact patients' perception of treatment effectiveness, the body of evidence, text quality, and hypothetical treatment intention. The experiment also examines whether there is an additive effect when multiple sources of uncertainty are communicated (Q3).

METHODS

We developed 8 variations of a research summary set in a hypothetical scenario for a treatment decision in the context of tinnitus. These were modified only in the degree of uncertainty relating to the evidence of the presented treatment. We recruited members of the German public from a Web-based research panel and randomized them to one of 8 variations of the research summary to examine the 3 research questions. The trial was only open to the members of the research panel. The outcomes are perception of the effectiveness of the treatment (primary), certainty in the judgement of treatment effectiveness, perception of the body of evidence relating to the treatment, text quality, and decisional intention (secondary). Outcomes were self-assessed. We aimed to recruit 1500 participants to the trial. The recruitment and data collection was fully automated. Ethical approval was waivered by an ethics committee because of the negligible risk to participants.

RESULTS

This protocol is retrospectively published in its original format. In the meantime, the trial was set up and the data collection was completed. Data collection was conducted in May 2018. A total of 1727 eligible panel members were enrolled.

CONCLUSIONS

We aim to publish the results in a peer-reviewed journal by the end of 2019. In addition, results will be presented at conferences and disseminated among developers of guidance for the development of evidence-based health information and decision aids.

TRIAL REGISTRATION

German Clinical Trials Register DRKS00015911; https://www.drks.de/drks_web/navigate.do? navigationId=trial.HTML&TRIAL_ID=DRKS00015911 (Archived by WebCite at http://www.webcitation.org/77zyZTGzk).

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/13425.

Numeracy and Understanding of Quantitative Aspects of Predictive Models: A Pilot Study

BACKGROUND

The assessment of user preferences for performance characteristics of patient-oriented clinical prediction models is lacking. It is unknown if complex statistical aspects of prediction models are readily understandable by a general audience.

OBJECTIVE

A pilot study was conducted among nonclinical audiences to determine the feasibility of interpreting statistical concepts that describe the performance of prediction models.

METHODS

We conducted a cross-sectional electronic survey using the Amazon Mechanical Turk platform. The survey instrument included educational modules about predictive models, sensitivity, specificity, and confidence intervals (CIs). Follow-up questions tested participants' abilities to interpret these characteristics with both verbatim and gist knowledge. Objective and subjective numeracy were assessed using previously validated instruments. We also tested understanding of these concepts when embedded in a sample discrete choice experiment task to establish feasibility for future elicitation of preferences using a discrete choice experiment design. Multivariable linear regression was used to identify factors associated with correct interpretation of statistical concepts.

RESULTS

Among 534 respondents who answered all nine questions, the mean correct responses was 95.9% (95% CI, 93.8-97.4) for sensitivity, 93.1% (95% CI, 90.5-95.0) for specificity, and 86.6% (95% CI, 83.3-89.3) for CIs. Verbatim interpretation was high for all concepts, but significantly higher than gist only for CIs (p < 0.001). Scores on each discrete choice experiment tasks were slightly lower in each category. Both objective and subjective numeracy were positively associated with an increased proportion of correct responses (p < 0.001).

CONCLUSION

These results suggest that a nonclinical audience can interpret quantitative performance measures of predictive models with very high accuracy. Future development of patient-facing clinical prediction models can feasibly incorporate patient preferences for model features into their development.

Dumping the information bucket: A qualitative study of clinicians caring for patients with early stage non-small cell lung cancer

OBJECTIVE

To evaluate the quality of patient-clinician communication and shared decision making (SDM) when two disparate treatments for early stage non-small cell lung cancer (NSCLC) are discussed.

METHODS

We conducted a qualitative study to evaluate the experiences of 20 clinicians caring for patients with clinical Stage I NSCLC prior to treatment, focusing on communication practices. We used directed content analysis and a patient-centered communication theoretical model to guide understanding of communication strategies.

RESULTS

All clinicians expressed the importance of providing information, especially for mitigating patient worry, despite recognition that patients recall only a small amount of the information given. When patients expressed distress, clinicians exhibited empathy but preferred to provide more information in order to address patient concerns. Most clinicians reported practicing SDM, however, they also reported not clearly eliciting patient preferences and values, a key part of SDM.

CONCLUSION

Communication with patients about treatment options for early stage NSCLC primary includes information giving. We found that only a few communication domains associated with SDM occurred regularly, and SDM may not be necessary in this clinical context.

PRACTICE IMPLICATIONS

Clinicians may need to incorporate nurse navigators or more written materials for effectively discussing potentially equivalent treatment options with their patients.