The effects of quality of evidence communication on perception of public health information about COVID-19: Two randomised controlled trials

What Affects Perceived Trustworthiness of Online Medical Information and Subsequent Treatment Decision Making? Randomized Trials on the Role of Uncertainty and Institutional Cues

Background. Online, algorithmically driven prognostic tools are increasingly important in medical decision making. Institutions developing such tools need to be able to communicate the precision and accuracy of the information in a trustworthy manner, and so many attempt to communicate uncertainties but also use institutional logos to underscore their trustworthiness. Bringing together theories on trust, uncertainty, and psychological distance in a novel way, we tested whether and how the communication of uncertainty and the presence of institutional logos together affected trust in medical information, the prognostic tool itself, and treatment decisions. Methods. A pilot and 2 online experiments in which UK (experiment 1) and worldwide (experiment 2) participants (Ntotal = 4,724) were randomized to 1 of 12 arms in a 3 (uncertainty cue) × 4 (institutional cue) between-subjects design. The stimulus was based on an existing medical prognostic tool. Results. Institutional trust was consistently associated with trust in the prognostic tool itself, while uncertainty information had no consistent effect. Institutional trust predicted the amount of weight participants reported placing on institutional endorsements in their decision making and the likelihood of switching from passive to active treatment in a hypothetical scenario. There was also a significant effect of psychological distance to (perceived hypotheticality of) the scenario. Conclusions/Implications. These results underline the importance of institutions demonstrating trustworthiness and building trust with their users. They also suggest that users tend to be insensitive to communications of uncertainty and that communicators may need to be highly explicit when attempting to warn of low precision or quality of evidence. The effect of the perceived hypotheticality of the scenario underscores the importance of realistic decision-making scenarios for studies and the role of familiarity with the decision dilemma generally.

Highlights

In a world where information for medical decision making is increasingly going to be provided through digital, online tools, institutions providing such tools need guidance on how best to communicate about their trustworthiness and precision.We find that people are fairly insensitive to cues designed to communicate uncertainty around the outputs of such tools. Even putting "ATTENTION" in bold font or explicitly pointing out the weaknesses in the data did not appear to affect people's decision making using the tool's outputs. Institutions should take note, and further work is required to determine how best to communicate uncertainty in a way that elicits appropriate caution in lay users.People were much more sensitive to institutional logos associated with the outputs. Generalized institutional trust (rather than trust in the specific institution whose logo was shown) was associated with how trustworthy, accurate, and reliable the tool, its algorithm, and the numbers it produced were perceived to be. This underscores the role of societal trust in institutions at large.Finally, as a note to researchers, we found a significant effect of how hypothetical or believable participants felt the experimental scenario was. This is a variable that seems rarely controlled for in studies and yet played as much of a role as some of our variables of interest, so we suggest that it is measured in future experiments.

The effects of communicating uncertainty around statistics, on public trust

Uncertainty around statistics is inevitable. However, communicators of uncertain statistics, particularly in high-stakes and potentially political circumstances, may be concerned that presenting uncertainties could undermine the perceived trustworthiness of the information or its source. In a large survey experiment (Study 1; N = 10 519), we report that communicating uncertainty around present COVID-19 statistics in the form of a numeric range (versus no uncertainty) may lead to slightly lower perceived trustworthiness of the number presented but has no impact on perceived trustworthiness of the source of the information. We also show that this minimal impact of numeric uncertainty on trustworthiness is also present when communicating future, projected COVID-19 statistics (Study 2; N = 2,309). Conversely, we find statements about the mere existence of uncertainty, without quantification, can reduce both perceived trustworthiness of the numbers and of their source. Our findings add to others suggesting that communicators can be transparent about statistical uncertainty without undermining their credibility as a source but should endeavour to provide a quantification, such as a numeric range, where possible.

Communication of anticancer drug benefits and related uncertainties to patients and clinicians: document analysis of regulated information on prescription drugs in Europe

OBJECTIVE

To evaluate the frequency with which relevant and accurate information about the benefits and related uncertainties of anticancer drugs are communicated to patients and clinicians in regulated information sources in Europe.

DESIGN

Document content analysis.

SETTING

European Medicines Agency.

PARTICIPANTS

Anticancer drugs granted a first marketing authorisation by the European Medicines Agency, 2017-19.

MAIN OUTCOME MEASURES

Whether written information on a product addressed patients' commonly asked questions about: who and what the drug is used for; how the drug was studied; types of drug benefit expected; and the extent of weak, uncertain, or missing evidence for drug benefits. Information on drug benefits in written sources for clinicians (summaries of product characteristics), patients (patient information leaflets), and the public (public summaries) was compared with information reported in regulatory assessment documents (European public assessment reports).

RESULTS

29 anticancer drugs that received a first marketing authorisation for 32 separate cancer indications in 2017-19 were included. General information about the drug (including information on approved indications and how the drug works) was frequently reported across regulated information sources aimed at both clinicians and patients. Nearly all summaries of product characteristics communicated full information to clinicians about the number and design of the main studies, the control arm (if any), study sample size, and primary measures of drug benefit. None of the patient information leaflets communicated information to patients about how drugs were studied. 31 (97%) summaries of product characteristics and 25 (78%) public summaries contained information about drug benefits that was accurate and consistent with information in regulatory assessment documents. The presence or absence of evidence that a drug extended survival was reported in 23 (72%) summaries of product characteristics and four (13%) public summaries. None of the patient information leaflets communicated information about the drug benefits that patients might expect based on study findings. Scientific concerns about the reliability of evidence on drug benefits, which were raised by European regulatory assessors for almost all drugs in the study sample, were rarely communicated to clinicians, patients, or the public.

CONCLUSIONS

The findings of this study highlight the need to improve the communication of the benefits and related uncertainties of anticancer drugs in regulated information sources in Europe to support evidence informed decision making by patients and their clinicians.

Multiple Forecast Visualizations (MFVs): Trade-offs in Trust and Performance in Multiple COVID-19 Forecast Visualizations

The prevalence of inadequate SARS-COV-2 (COVID-19) responses may indicate a lack of trust in forecasts and risk communication. However, no work has empirically tested how multiple forecast visualization choices impact trust and task-based performance. The three studies presented in this paper ( N=1299) examine how visualization choices impact trust in COVID-19 mortality forecasts and how they influence performance in a trend prediction task. These studies focus on line charts populated with real-time COVID-19 data that varied the number and color encoding of the forecasts and the presence of best/worst-case forecasts. The studies reveal that trust in COVID-19 forecast visualizations initially increases with the number of forecasts and then plateaus after 6-9 forecasts. However, participants were most trusting of visualizations that showed less visual information, including a 95% confidence interval, single forecast, and grayscale encoded forecasts. Participants maintained high trust in intervals labeled with 50% and 25% and did not proportionally scale their trust to the indicated interval size. Despite the high trust, the 95% CI condition was the most likely to evoke predictions that did not correspond with the actual COVID-19 trend. Qualitative analysis of participants' strategies confirmed that many participants trusted both the simplistic visualizations and those with numerous forecasts. This work provides practical guides for how COVID-19 forecast visualizations influence trust, including recommendations for identifying the range where forecasts balance trade-offs between trust and task-based performance.

Transparent communication of evidence does not undermine public trust in evidence

Does clear and transparent communication of risks, benefits, and uncertainties increase or undermine public trust in scientific information that people use to guide their decision-making? We examined the impact of reframing messages written in traditional persuasive style to align instead with recent "evidence communication" principles, aiming to inform decision-making: communicating a balance of risks and benefits, disclosing uncertainties and evidence quality, and prebunking misperceptions. In two pre-registered experiments, UK participants read either a persuasive message or a balanced and informative message adhering to evidence communication recommendations about COVID-19 vaccines (Study 1) or nuclear power plants (Study 2). We find that balanced messages are either perceived as trustworthy as persuasive messages (Study 1), or more so (Study 2). However, we note a moderating role of prior beliefs such that balanced messages were consistently perceived as more trustworthy among those with negative or neutral prior beliefs about the message content. We furthermore note that participants who had read the persuasive message on nuclear power plants voiced significantly stronger support for nuclear power than those who had read the balanced message, despite rating the information as less trustworthy. There was no difference in vaccination intentions between groups reading the different vaccine messages.

Adherence of those at low risk of disease to public health measures during the COVID-19 pandemic: A qualitative study

Public health measures (PHMs) proactively and reactively reduce the spread of disease. While these measures target individual behaviour, they require broad adherence to be effective. Consequently, the World Health Organization issued a special appeal to young adults, a known non-adherent population, for increased adherence with COVID-19 guidelines. However, little is known about why these low-risk individuals do or do not adhere to PHMs. This study investigates why young adults in a low-risk setting adhered to PHMs implemented during the COVID-19 pandemic. A qualitative research approach was chosen to gain an in-depth understanding of participants’ thoughts and experiences related to PHM adherence. Semi-structured interviews were conducted in April-May 2021 with 30 young adults living in Prince Edward Island (PEI), the province with the lowest COVID-19 case rate in Canada at that time. Thematic analysis was used to create a codebook based on the Theoretical Domains Framework, which was then inductively modified. The analysis identified eight themes that explained the adherence of young adults: (1) clear, purpose-driven adherence rationale, (2) developing trust in the local leadership, (3) adapting to novel measures, (4) manageable disruption, (5) adhering to reduce anxiety, (6) collective duty towards one’s community, (7) moral culpability and (8) using caution rather than compliance. Together, these themes demonstrate that young adults adhered to PHMs because of their sense of connection to their community, public health leadership, and concerns over stigma. We further argue that clear guidelines and communication from public health officials during both periods of high and low COVID-19 cases facilitate adherence. These findings are important for mitigating future public health emergencies as they explain why young adults, an important segment of the population whose adherence is critical to the success of PHMs, follow PHMs. Further, these findings can inform public health officials and other stakeholders aiming to develop successful adherence strategies.

The effects of communicating scientific uncertainty on trust and decision making in a public health context

Large-scale societal issues such as public health crises highlight the need to communicate scientific information, which is often uncertain, accurately to the public and policy makers. The challenge is to communicate the inherent scientific uncertainty — especially about the underlying quality of the evidence — whilst supporting informed decision making. Little is known about the effects that such scientific uncertainty has on people’s judgments of the information. In three experimental studies (total N=6,489), we investigate the influence of scientific uncertainty about the quality of the evidence on people’s perceived trustworthiness of the information and decision making. We compare the provision of high, low, and ambiguous quality-of-evidence indicators against providing no such cues. Results show an asymmetric relationship: people react more strongly to cues of low quality of evidence than they do to high quality of evidence compared to no cue. While responses to a cue of high quality of evidence are not significantly different from no cue; a cue of low or uncertain quality of evidence is accompanied by lower perceived trustworthiness and lower use of the information in decision making. Cues of uncertain quality of evidence have a similar effect to those of low quality. These effects do not change with the addition of a reason for the indicated quality level. Our findings shed light on the effects of the communication of scientific uncertainty on judgment and decision making, and provide insights for evidence-based communications and informed decision making for policy makers and the public.

Health communication in and out of public health emergencies: to persuade or to inform?

Much health communication during the COVID-19 pandemic has been designed to persuade people more than to inform them. For example, messages like “masks save lives” are intended to compel people to wear face masks, not to enable them to make an informed decision about whether to wear a face mask or to understand the justification for a mask mandate. Both persuading people and informing them are reasonable goals for health communication. However, those goals can sometimes be in conflict. In this article, we discuss potential conflicts between seeking to persuade or to inform people, the use of spin to persuade people, the ethics of persuasion, and implications for health communication in the context of the pandemic and generally. Decisions to persuade people rather than enable them to make an informed choice may be justified, but the basis for those decisions should be transparent and the evidence should not be distorted. We suggest nine principles to guide decisions by health authorities about whether to try to persuade people.

What drives compliance with COVID-19 measures over time? Explaining changing impacts with Goal Framing Theory

The COVID-19 pandemic provides a unique opportunity to study which factors drive compliance and how the evolving context in society -virus fluctuations and changing government measures - changes the impact of these factors. Extant literature lists many factors that drive compliance - notably enforcement, trust, legitimacy. Most of these studies, however, do not look across time: whether a changing context for citizens changes the impact of factors driving compliance. In this study, we use Lindenberg's Goal Framing Theory to explain the dynamics of these drivers of compliance during the COVID-19 pandemic. We formulate hypotheses for pro-socialness, trust in government, observed respect for rules, rule effectiveness, rule appropriateness, fear of COVID-19 (severity and proximity), opportunities for pleasure and happiness, as well as worsened income position. We test our hypotheses with data collected at three different moments during the beginning of the COVID-19 crisis in Flanders, Belgium. Findings show that over time the constellations of factors that drive compliance change and, later in the pandemic, more distinct groups of citizens with different motivations to comply are identified. The overall conclusion is that the voluntary basis for compliance becomes more fragile over time, with a more differentiated pattern of drivers of compliance emerging. Public policy and communication need to adapt to these changes over time and address different groups of citizens.