The effects of presenting imprecise probabilities in intelligence forecasts

When evidence changes: Communicating uncertainty protects against a loss of trust

Scientific findings can be overturned when new evidence arises. This study examines how communicating and explaining uncertainty around scientific findings affect trust in the communicator when findings change. In an online experiment (N = 800, convenience sample), participants read a fictitious statement from a public health authority announcing that there was no link between a new COVID-19 vaccine and heart muscle inflammation. The authority communicated (1) no uncertainty, (2) uncertainty without giving a reason, (3) uncertainty due to imprecision, or (4) uncertainty due to incomplete accounting of patients. Participants were then informed that the authority's statement was no longer correct as new data showed a link between the vaccine and heart muscle inflammation. Participants rated the authority's trustworthiness before and after the evidence update. Our findings indicate that communicating uncertainty buffers against a loss of trust when evidence changes. Moreover, explaining uncertainty does not appear to harm trust.

Probability and informed consent

In this paper, we illustrate some serious difficulties involved in conveying information about uncertain risks and securing informed consent for risky interventions in a clinical setting. We argue that in order to secure informed consent for a medical intervention, physicians often need to do more than report a bare, numerical probability value. When probabilities are given, securing informed consent generally requires communicating how probability expressions are to be interpreted and communicating something about the quality and quantity of the evidence for the probabilities reported. Patients may also require guidance on how probability claims may or may not be relevant to their decisions, and physicians should be ready to help patients understand these issues.

The effects of communicating uncertainty around statistics, on public trust

Uncertainty around statistics is inevitable. However, communicators of uncertain statistics, particularly in high-stakes and potentially political circumstances, may be concerned that presenting uncertainties could undermine the perceived trustworthiness of the information or its source. In a large survey experiment (Study 1; N = 10 519), we report that communicating uncertainty around present COVID-19 statistics in the form of a numeric range (versus no uncertainty) may lead to slightly lower perceived trustworthiness of the number presented but has no impact on perceived trustworthiness of the source of the information. We also show that this minimal impact of numeric uncertainty on trustworthiness is also present when communicating future, projected COVID-19 statistics (Study 2; N = 2,309). Conversely, we find statements about the mere existence of uncertainty, without quantification, can reduce both perceived trustworthiness of the numbers and of their source. Our findings add to others suggesting that communicators can be transparent about statistical uncertainty without undermining their credibility as a source but should endeavour to provide a quantification, such as a numeric range, where possible.

Negative consequences of failing to communicate uncertainties during a pandemic: an online randomised controlled trial on COVID-19 vaccines

OBJECTIVE

To examine the impact of the government communicating uncertainties relating to COVID-19 vaccine effectiveness on vaccination intention and trust after people are exposed to conflicting information.

DESIGN

Experimental design where participants were randomly allocated to one of two groups.

SETTING

Online.

PARTICIPANTS

328 adults from a UK research panel.

INTERVENTION

Participants received either certain or uncertain communications from a government representative about COVID-19 vaccine effectiveness, before receiving conflicting information about effectiveness.

MAIN OUTCOME MEASURES

Vaccination intention and trust in government.

RESULTS

Compared with those who received the uncertain announcement from the government, participants who received the certain announcement reported a greater loss of vaccination intention (d=0.34, 95% CI (0.12 to 0.56), p=0.002) and trust (d=0.34, 95% CI (0.12 to 0.56), p=0.002) after receiving conflicting information.

CONCLUSIONS

Communicating with certainty about COVID-19 vaccines reduces vaccination intention and trust if conflicting information arises, whereas communicating uncertainties can protect people from the negative impact of exposure to conflicting information. There are likely to be other factors affecting vaccine intentions, which we do not account for in this study.

TRIAL REGISTRATION NUMBER

Open Science Framework: https://osf.io/c73px/.

Effects of communicating uncertainty descriptions in hazard identification, risk characterization, and risk protection

Uncertainty is a crucial issue for any risk assessment. Consequently, it also poses crucial challenges for risk communications. Many guidebooks advise reporting uncertainties in risk assessments, expecting that the audience will appreciate this disclosure. However, the empirical evidence about the effects of uncertainty reporting is sparse and inconclusive. Therefore, based on examples of potential health risks of electromagnetic fields (EMF), three experiments were conducted analysing the effects of communicating uncertainties separately for hazard identification, risk characterisation and risk protection. The setups aimed to explore how reporting and how explaining of uncertainty affects dependent variables such as risk perception, perceived competence of the risk assessors, and trust in risk management. Each of the three experiments used a 2x2 design with a first factor presenting uncertainty descriptions (as used in public controversies on EMF related health effects) or describing a certainty conditions; and a second factor explaining the causes of uncertainties (by pointing at knowledge gaps) or not explaining them. The study results indicate that qualitative uncertainty descriptions regarding hazard identification reduce the confidence in the professional competencies of the assessors. In contrast, a quantitative uncertainty description in risk characterisation-regarding the magnitude of the risk-does not affect any of the dependent variables. Concerning risk protection, trust in exposure limit values is not affected by qualitative uncertainty information. However, the qualitative description of uncertainty regarding the adequacy of protection amplifies fears. Furthermore, explaining this uncertainty results in lower text understandability.

Cultivating credibility with probability words and numbers

Recent research suggests that communicating probabilities numerically rather than verbally benefits forecasters’ credibility. In two experiments, we tested the reproducibility of this communication-format effect. The effect was replicated under comparable conditions (low-probability, inaccurate forecasts), but it was reversed for low-probability accurate forecasts and eliminated for high-probability forecasts. Experiment 2 further showed that verbal probabilities convey implicit recommendations more clearly than probability information, whereas numeric probabilities do the opposite. Descriptively, the findings indicate that the effect of probability words versus numbers on credibility depends on how these formats convey directionality differently, how directionality implies recommendations even when none are explicitly given, and how such recommendations correspond with outcomes. Prescriptively, we propose that experts distinguish forecasts from advice, using numeric probabilities for the former and well-reasoned arguments for the latter.

Guidance on Communication of Uncertainty in Scientific Assessments

This document provides guidance for communicators on how to communicate the various expressions of uncertainty described in EFSA's document: 'Guidance on uncertainty analysis in scientific assessments'. It also contains specific guidance for assessors on how best to report the various expressions of uncertainty. The document provides a template for identifying expressions of uncertainty in scientific assessments and locating the specific guidance for each expression. The guidance is structured according to EFSA's three broadly defined categories of target audience: 'entry', 'informed' and 'technical' levels. Communicators should use the guidance for entry and informed audiences, while assessors should use the guidance for the technical level. The guidance was formulated using evidence from the scientific literature, grey literature and two EFSA research studies, or based on judgement and reasoning where evidence was incomplete or missing. The limitations of the evidence sources inform the recommendations for further research on uncertainty communication.

Effects of Acknowledging Uncertainty about Earthquake Risk Estimates on San Francisco Bay Area Residents' Beliefs, Attitudes, and Intentions

We test here the risk communication proposition that explicit expert acknowledgment of uncertainty in risk estimates can enhance trust and other reactions. We manipulated such a scientific uncertainty message, accompanied by probabilities (20%, 70%, implicit ["will occur"] 100%) and time periods (10 or 30 years) in major (≥magnitude 8) earthquake risk estimates to test potential effects on residents potentially affected by seismic activity on the San Andreas fault in the San Francisco Bay Area (n = 750). The uncertainty acknowledgment increased belief that these specific experts were more honest and open, and led to statistically (but not substantively) significant increases in trust in seismic experts generally only for the 20% probability (vs. certainty) and shorter versus longer time period. The acknowledgment did not change judged risk, preparedness intentions, or mitigation policy support. Probability effects independent of the explicit admission of expert uncertainty were also insignificant except for judged risk, which rose or fell slightly depending upon the measure of judged risk used. Overall, both qualitative expressions of uncertainty and quantitative probabilities had limited effects on public reaction. These results imply that both theoretical arguments for positive effects, and practitioners' potential concerns for negative effects, of uncertainty expression may have been overblown. There may be good reasons to still acknowledge experts' uncertainties, but those merit separate justification and their own empirical tests.

The Role of Type and Source of Uncertainty on the Processing of Climate Models Projections

Scientists agree that the climate is changing due to human activities, but there is less agreement about the specific consequences and their timeline. Disagreement among climate projections is attributable to the complexity of climate models that differ in their structure, parameters, initial conditions, etc. We examine how different sources of uncertainty affect people's interpretation of, and reaction to, information about climate change by presenting participants forecasts from multiple experts. Participants viewed three types of sets of sea-level rise projections: (1) precise, but conflicting; (2) imprecise, but agreeing, and (3) hybrid that were both conflicting and imprecise. They estimated the most likely sea-level rise, provided a range of possible values and rated the sets on several features - ambiguity, credibility, completeness, etc. In Study 1, everyone saw the same hybrid set. We found that participants were sensitive to uncertainty between sources, but not to uncertainty about which model was used. The impacts of conflict and imprecision were combined for estimation tasks and compromised for feature ratings. Estimates were closer to the experts' original projections, and sets were rated more favorably under imprecision. Estimates were least consistent with (narrower than) the experts in the hybrid condition, but participants rated the conflicting set least favorably. In Study 2, we investigated the hybrid case in more detail by creating several distinct interval sets that combine conflict and imprecision. Two factors drive perceptual differences: overlap - the structure of the forecast set (whether intersecting, nested, tangent, or disjoint) - and asymmetry - the balance of the set. Estimates were primarily driven by asymmetry, and preferences were primarily driven by overlap. Asymmetric sets were least consistent with the experts: estimated ranges were narrower, and estimates of the most likely value were shifted further below the set mean. Intersecting and nested sets were rated similarly to imprecision, and ratings of disjoint and tangent sets were rated like conflict. Our goal was to determine which underlying factors of information sets drive perceptions of uncertainty in consistent, predictable ways. The two studies lead us to conclude that perceptions of agreement require intersection and balance, and overly precise forecasts lead to greater perceptions of disagreement and a greater likelihood of the public discrediting and misinterpreting information.

Seeing What You Want to See: How Imprecise Uncertainty Ranges Enhance Motivated Reasoning

In this article, we consider a novel criterion for evaluating representations of uncertainty ranges, namely, the extent to which a representation enhances motivated reasoning. In two studies, we show that perceptions of the distribution underlying ambiguous numerical ranges are affected by the motivations and worldviews of end users. This motivated reasoning effect remained after controlling for objective numeracy and fluid intelligence but was attenuated when the correct interpretation was made clear. We suggest that analysts and communicators explicitly consider the potential for motivated evaluation when evaluating uncertainty displays.

Communicating uncertainty in seasonal and interannual climate forecasts in Europe

Across Europe, organizations in different sectors are sensitive to climate variability and change, at a range of temporal scales from the seasonal to the interannual to the multi-decadal. Climate forecast providers face the challenge of communicating the uncertainty inherent in these forecasts to these decision-makers in a way that is transparent, understandable and does not lead to a false sense of certainty. This article reports the findings of a user-needs survey, conducted with 50 representatives of organizations in Europe from a variety of sectors (e.g. water management, forestry, energy, tourism, health) interested in seasonal and interannual climate forecasts. We find that while many participating organizations perform their own 'in house' risk analysis most require some form of processing and interpretation by forecast providers. However, we also find that while users tend to perceive seasonal and interannual forecasts to be useful, they often find them difficult to understand, highlighting the need for communication formats suitable for both expert and non-expert users. In addition, our results show that people tend to prefer familiar formats for receiving information about uncertainty. The implications of these findings for both the providers and users of climate information are discussed.

Does Introducing Imprecision around Probabilities for Benefit and Harm Influence the Way People Value Treatments?

Background. Imprecision in estimates of benefits and harms around treatment choices is rarely described to patients. Variation in sampling error between treatment alternatives (e.g., treatments have similar average risks, but one treatment has a larger confidence interval) can result in patients failing to choose the option that is best for them. The aim of this study is to use a discrete choice experiment to describe how 2 methods for conveying imprecision in risk influence people’s treatment decisions. Methods. We randomized a representative sample of the Canadian general population to 1 of 3 surveys that sought choices between hypothetical treatments for rheumatoid arthritis based on different levels of 7 attributes: route and frequency of administration, chance of benefit, serious and minor side effects and life expectancy, and imprecision in benefit and side-effect estimates. The surveys differed in the way imprecision was described: 1) no imprecision, 2) quantitative description based on a range with a visual graphic, and 3) qualitative description simply describing the confidence in the evidence. Results. The analyzed data were from 2663 respondents. Results suggested that more people understood imprecision when it was described qualitatively (88%) versus quantitatively (68%). Respondents who appeared to understand imprecision descriptions placed high value on increased precision regarding the actual benefits and harms of treatment, equivalent to the value placed on the information about the probability of serious side effects. Both qualitative and quantitative methods led to small but significant increases in decision uncertainty for choosing any treatment. Limitations included some issues in defining understanding of imprecision and the use of an internet survey of panel members. Conclusions. These findings provide insight into how conveying imprecision information influences patient treatment choices.

At Home on the Range? Lay Interpretations of Numerical Uncertainty Ranges

Numerical uncertainty ranges are often used to convey the precision of a forecast. In three studies, we examined how users perceive the distribution underlying numerical ranges and test specific hypotheses about the display characteristics that affect these perceptions. We discuss five primary conclusions from these studies: (1) substantial variation exists in how people perceive the distribution underlying numerical ranges; (2) distributional perceptions appear similar whether the uncertain variable is a probability or an outcome; (3) the variation in distributional perceptions is due in part to individual differences in numeracy, with more numerate individuals more likely to perceive the distribution as roughly normal; (4) the variation is also due in part to the presence versus absence of common cues used to convey the correct interpretation (e.g., including a best estimate increases perceptions that the distribution is roughly normal); and (5) simple graphical representations can decrease the variance in distributional perceptions. These results point toward significant opportunities to improve uncertainty communication in climate change and other domains.

Unquestioned answers or unanswered questions: beliefs about science guide responses to uncertainty in climate change risk communication

In two experimental studies we investigated the effect of beliefs about the nature and purpose of science (classical vs. Kuhnian models of science) on responses to uncertainty in scientific messages about climate change risk. The results revealed a significant interaction between both measured (Study 1) and manipulated (Study 2) beliefs about science and the level of communicated uncertainty on willingness to act in line with the message. Specifically, messages that communicated high uncertainty were more persuasive for participants who shared an understanding of science as debate than for those who believed that science is a search for absolute truth. In addition, participants who had a concept of science as debate were more motivated by higher (rather than lower) uncertainty in climate change messages. The results suggest that achieving alignment between the general public's beliefs about science and the style of the scientific messages is crucial for successful risk communication in science. Accordingly, rather than uncertainty always undermining the effectiveness of science communication, uncertainty can enhance message effects when it fits the audience's understanding of what science is.