Communicating uncertainty using words and numbers

Do claims about certainty make estimates less certain?

Predictions and estimates are sometimes qualified as certain. This epistemic marker occupies a privileged position at the top of scales of verbal probability expressions, reflecting probabilities close to 1. But such statements have rarely been compared to plain, unqualified statements in which certainty is not mentioned. We examined in nine studies (N = 2784) whether statements explicitly claimed to be certain are perceived as (1) more (or less) credible, (2) more (or less) precise, and (3) more (or less) strongly based upon evidence, compared to plain, unmarked declarative statements. We find, in apparent contrast with assumptions made by the standard scales, that "certain" are often judged to be less trustworthy, less reliable, and held with lower confidence than unmarked statements. Plain, declarative statements are further assumed to be more precise, while certainty implies that more extreme outcomes are possible. When it is certain that Henry made four errors, it is clear he did not commit less than four, but he might have committed five errors or more. Thus certainty can indicate lower bounds of an interval whose upper bounds are not defined, and certainty statements are consequently more ambiguous than estimates that do not mention certainty. At least-interpretations of certainty affect the interpretation of options in risky choice problems, where "200 lives will be saved" was deemed by a majority to mean exactly 200, while "it is certain that 200 will be saved", could mean 200-600 lives. We also find that credibility is affected by type of certainty, with impersonal certainty ("it is certain") perceived to be more accurate and persuasive than personal certainty ("I am certain"), especially in predictions of future events. Moreover, mentions of certainty can reveal that that a speaker's estimate is based on subjective judgments and guesswork rather than upon objective evidence. These findings have implications for communication. Estimates can appear more consensual when claims of certainty are omitted. To convey certainty it may be better not to mention that one is certain.

Analysis of implicit and explicit uncertainties in QSAR prediction of chemical toxicity: A case study of neurotoxicity

Although uncertainties expressed in texts within QSAR studies can guide quantitative uncertainty estimations, they are often overlooked during uncertainty analysis. Using neurotoxicity as an example, this study developed a method to support analysis of implicitly and explicitly expressed uncertainties in QSAR modeling studies. Text content analysis was employed to identify implicit and explicit uncertainty indicators, whereafter uncertainties within the indicator-containing sentences were identified and systematically categorized according to 20 uncertainty sources. Our results show that implicit uncertainty was more frequent within most uncertainty sources (13/20), while explicit uncertainty was more frequent in only three sources, indicating that uncertainty is predominantly expressed implicitly in the field. The most highly cited sources included Mechanistic plausibility, Model relevance and Model performance, suggesting they constitute sources of most concern. The fact that other sources like Data balance were not mentioned, although it is recognized in the broader QSAR literature as an area of concern, demonstrates that the output from the type of analysis conducted here must be interpreted in the context of the broader QSAR literature before conclusions are drawn. Overall, the method established here can be applied in other QSAR modeling contexts and ultimately guide efforts targeted towards addressing the identified uncertainty sources.

Severity influences categorical likelihood communications: A case study with Southeast Asian weather forecasters

Risk assessments are common in multiple domains, from finance to medicine. They require evaluating an event's potential severity and likelihood. We investigate the possible dependence of likelihood and severity within the domain of impact-based weather forecasting (IBF), following predictions derived from considering asymmetric loss functions. In a collaboration between UK psychologists and partners from four meteorological organisations in Southeast Asia, we conducted two studies (N = 363) eliciting weather warnings from forecasters. Forecasters provided warnings denoting higher likelihoods for high severity impacts than low severity impacts, despite these impacts being described as having the same explicit numerical likelihood of occurrence. This 'Severity effect' is pervasive, and we find it can have a continued influence even for an updated forecast. It is additionally observed when translating warnings made on a risk matrix to numerical probabilities.

When leaders disclose uncertainty: Effects of expressing internal and external uncertainty about a decision

It is generally assumed that decision-makers appear more competent and trustworthy when exuding confidence in their choices. However, many decisions are by their nature uncertain. Is it possible for a decision-maker to admit uncertainty and still be trusted? We propose that the communicated type of uncertainty may matter. Internal uncertainty, which signals lack of knowledge or a low degree of belief, may be viewed more negatively than external uncertainty, which is associated with randomness and complexity. The results of a series of experiments suggested that people viewed leaders as more competent when they expressed uncertainty about a decision in external ("It is uncertain") rather than internal terms ("I am uncertain"), overall effect size d = 0.45 [0.16, 0.74]. Paradoxically, when asked directly, participants expressed that leaders should be open about uncertainty rather than exuding confidence and downplaying uncertainty. A final study suggested that decision makers were more willing to reveal uncertainty about a choice to others when they perceived the uncertainty as more external and less internal and expected more positive and fewer negative consequences from expressing external rather than internal uncertainty.

A comparison of human and GPT-4 use of probabilistic phrases in a coordination game

English speakers use probabilistic phrases such as likely to communicate information about the probability or likelihood of events. Communication is successful to the extent that the listener grasps what the speaker means to convey and, if communication is successful, individuals can potentially coordinate their actions based on shared knowledge about uncertainty. We first assessed human ability to estimate the probability and the ambiguity (imprecision) of twenty-three probabilistic phrases in a coordination game in two different contexts, investment advice and medical advice. We then had GPT-4 (OpenAI), a Large Language Model, complete the same tasks as the human participants. We found that GPT-4's estimates of probability both in the Investment and Medical Contexts were as close or closer to that of the human participants as the human participants' estimates were to one another. However, further analyses of residuals disclosed small but significant differences between human and GPT-4 performance. Human probability estimates were compressed relative to those of GPT-4. Estimates of probability for both the human participants and GPT-4 were little affected by context. We propose that evaluation methods based on coordination games provide a systematic way to assess what GPT-4 and similar programs can and cannot do.

The effects of communicating uncertainty around statistics, on public trust

Uncertainty around statistics is inevitable. However, communicators of uncertain statistics, particularly in high-stakes and potentially political circumstances, may be concerned that presenting uncertainties could undermine the perceived trustworthiness of the information or its source. In a large survey experiment (Study 1; N = 10 519), we report that communicating uncertainty around present COVID-19 statistics in the form of a numeric range (versus no uncertainty) may lead to slightly lower perceived trustworthiness of the number presented but has no impact on perceived trustworthiness of the source of the information. We also show that this minimal impact of numeric uncertainty on trustworthiness is also present when communicating future, projected COVID-19 statistics (Study 2; N = 2,309). Conversely, we find statements about the mere existence of uncertainty, without quantification, can reduce both perceived trustworthiness of the numbers and of their source. Our findings add to others suggesting that communicators can be transparent about statistical uncertainty without undermining their credibility as a source but should endeavour to provide a quantification, such as a numeric range, where possible.

Evaluation of a decision support system using Bayesian network modeling in an applied Multi-INT surveillance environment

Sensemaking and decision-making are fundamental components of applied Intelligence, Surveillance, and Reconnaissance (ISR). Analysts acquire information from multiple sources over a period of hours, days, or even over the scale of months or years, that must be interpreted and integrated to predict future adversarial events. Sensemaking is essential for developing an appropriate mental model that will lead to accurate predictions sooner. Decision Support Systems (DSS) are one proposed solution to improve analyst decision-making outcomes by leveraging computers to conduct calculations that may be difficult for human operators and provide recommendations. In this study, we tested two simulated DSS that were informed by a Bayesian Network Model as a potential prediction-assistive tool. Participants completed a simulated multi-day, multi-source intelligence task and were asked to make predictions regarding five potential outcomes on each day. Participants in both DSS conditions were able to converge on the correct solution significantly faster than the control group, and between 36-44% more of the sample was able to reach the correct conclusion. Furthermore, we found that a DSS representing projected outcome probabilities as numerical, rather than using verbal ordinal labels, were better able to differentiate which outcomes were extremely unlikely than the control group or verbal-probability DSS.

Age differences in the neural basis of decision-making under uncertainty

Humans globally are reaping the benefits of longer lives. Yet, longer life spans also require engaging with consequential but often uncertain decisions well into old age. Previous research has yielded mixed findings with regards to life span differences in how individuals make decisions under uncertainty. One factor contributing to the heterogeneity of findings is the diversity of paradigms that cover different aspects of uncertainty and tap into different cognitive and affective mechanisms. In this study, 175 participants (53.14% females, mean age = 44.9 years, SD = 19.0, age range = 16 to 81) completed functional neuroimaging versions of two prominent paradigms in this area, the Balloon Analogue Risk Task and the Delay Discounting Task. Guided by neurobiological accounts of age-related changes in decision-making under uncertainty, we examined age effects on neural activation differences in decision-relevant brain structures, and compared these across multiple contrasts for the two paradigms using specification curve analysis. In line with theoretical predictions, we find age differences in nucleus accumbens, anterior insula, and medial prefrontal cortex, but the results vary across paradigm and contrasts. Our results are in line with existing theories of age differences in decision making and their neural substrates, yet also suggest the need for a broader research agenda that considers how both individual and task characteristics determine the way humans deal with uncertainty.

Communicating uncertainty in national security intelligence: Expert and nonexpert interpretations of and preferences for verbal and numeric formats

Organizations in several domains including national security intelligence communicate judgments under uncertainty using verbal probabilities (e.g., likely) instead of numeric probabilities (e.g., 75% chance), despite research indicating that the former have variable meanings across individuals. In the intelligence domain, uncertainty is also communicated using terms such as low, moderate, or high to describe the analyst's confidence level. However, little research has examined how intelligence professionals interpret these terms and whether they prefer them to numeric uncertainty quantifiers. In two experiments (N = 481 and 624, respectively), uncertainty communication preferences of expert (n = 41 intelligence analysts in Experiment 1) and nonexpert intelligence consumers were elicited. We examined which format participants judged to be more informative and simpler to process. We further tested whether participants treated verbal probability and confidence terms as independent constructs and whether participants provided coherent numeric probability translations of verbal probabilities. Results showed that although most nonexperts favored the numeric format, experts were about equally split, and most participants in both samples regarded the numeric format as more informative. Experts and nonexperts consistently conflated probability and confidence. For instance, confidence intervals inferred from verbal confidence terms had a greater effect on the location of the estimate than the width of the estimate, contrary to normative expectation. Approximately one-fourth of experts and over one-half of nonexperts provided incoherent numeric probability translations for the terms likely and unlikely when the elicitation of best estimates and lower and upper bounds were briefly spaced by intervening tasks.

Dimensions of uncertainty communication: What is conveyed by verbal terms and numeric ranges

The paper reviews two strands of research on communication of uncertainty that usually have been investigated separately: (1) Probabilities attached to specific outcomes, and (2) Range judgments. Probabilities are sometimes expressed by verbal phrases ("rain is likely") and at other times in a numeric format ("70% chance of rain"), whereas range judgments describe the potential amounts expected ("1-4 mm of rain"). Examination of previous research shows that both descriptions convey, in addition to the strength of expectations, pragmatic information about the communicative situation. For instance, so-called verbal probability expressions (VPE), as likely, unlikely, a chance, or not certain give some, albeit vague, probabilistic information, but carry in addition an implicit message about the sources of uncertainty, the outcome's valence and severity, along with information about the speakers' attitudes and their communicative intentions. VPEs are directional by drawing attention either to an outcome's occurrence ("it is possible") or to its non-occurrence ("it is doubtful"). In this sense they may be more informative than numbers. Uncertainties about outcomes in a distribution (continuous quantities) are alternatively expressed as interval estimates. The width of such intervals can function as a cue to credibility and expertise. Incomplete, one-sided intervals, where only one boundary is stated, imply directionality. "More than 100 people" suggests a crowd, while "less than 200" implies a shortfall. As with VPEs, directionally positive intervals are more frequent, and perhaps more neutral than negative ones. To convey expectancies and uncertainty in a balanced way, communicators may have to alternate between complementary frames.