Intuitive theories of information: beliefs about the value of redundancy

Sensitivity to Evidential Dependencies in Judgments Under Uncertainty

According to Bayesian models of judgment, testimony from independent informants has more evidential value than dependent testimony. Three experiments investigated learners' sensitivity to this distinction. Each experiment used a social version of the balls‐and‐urns task, in which participants judged which of two urns was the most likely source of evidence presented by multiple informants. Informants either provided independent testimony based solely on their own observations or dependent‐sequential testimony that considered the testimonies of previous informants. Although participants updated their beliefs with additional evidence, this updating was generally insensitive to evidential dependency (Experiments 1 and 2). A notable exception was when individuals were separated according to their beliefs about the relative value of independent and sequential evidence. Those who viewed independent evidence as having greater value subsequently gave more weight to independent testimony in the balls‐and‐urns task (Experiment 3), in line with the predictions of a Bayesian model. Our findings suggest that only a minority of individuals conform to Bayesian predictions in the relative weighting of independent and dependent evidence in judgments under uncertainty.

The diversity principle and the evaluation of evidence

The diversity principle-the intuitive notion that diverse evidence is, all else equal, more persuasive, suggestive, confirmatory, or otherwise better than less varied sets of evidence-is a clear component of scientific practice and endorsed by scientists and philosophers alike. A great body of psychological research on people's understanding and application of the diversity principle exists, yet it remains divided into multiple, distinct research communities, which often come to conflicting conclusions. One reason for this is that the range of tasks and domains investigated is appropriately wide. Without a common understanding of what it means for evidence to be diverse, however, it is hard to compare what are at times diverging results. To address this, I review three perspectives from philosophy on what makes diverse evidence valuable. I will use the perspectives to frame results from psychology and assess whether people understand the value of diverse evidence on both an intuitive and explicit level. My conclusions have a leveled optimism: While people are generally aware of the value of diverse evidence, they often struggle to apply what they know. I argue this is because people do not assess the diversity of their evidence as a matter of course but rely on its intuitive diversity as a cue to its evidential diversity. When this cue is absent, people can overlook otherwise obvious problems with their evidence. This has potential consequences for how people seek out, evaluate, and understand evidence from a variety of domains, but leaves open the possibility that various interventions-such as education or reminders to attend to the quality of evidence-may increase people's application of what they know.'

Who are the "Real" Experts? The Debate Surrounding COVID-19 Health Risk Management: An Israeli Case Study

BACKGROUND

The uncertainty surrounding the COVID-19 crisis and the different approaches taken to manage it have triggered scientific controversies among experts. This study seeks to examine how the fragile nature of Israeli democracy accommodated differences of opinion between experts during the COVID-19 crisis.

OBJECTIVE

To map and analyze the discourse between experts surrounding issues that were the topic of scientific controversy. To examine the viewpoints of the public regarding the positions of the different experts.

METHODS AND SAMPLE

A sequential mixed study design. The qualitative research was a discourse analysis of 435 items that entailed mapping the voices of different experts regarding controversial topics. In the quantitative study, a total of 924 participants answered a questionnaire examining topics that engendered differences of opinion between the experts.

RESULTS

The results showed that there was no dialogue between opposition and coalition experts. Moreover, the coalition experts labeled the experts who criticized them as "coronavirus deniers" and "anti-vaxxers." The coalition changed its opinion on one issue only-the issue of lockdowns. When we asked the public how they see the scientific controversy between the coalition and the opposition experts, they expressed support for opposition policies on matters related to the implications of the lockdowns and to transparency, while supporting government policy mainly on topics related to vaccinations. The research findings also indicate that personal and socio-demographic variables can influence how the public responds to the debate between experts. The main differentiating variables were the personal attribute of conservatism, locus of control, age, and nationality.

CONCLUSION

Controversy must be encouraged to prevent misconceptions. The internal discourse in the committees that advise the government must be transparent, and coalition experts must be consistently exposed to the views of opposition experts, who must be free to voice their views without fear.

Dependencies in evidential reports: The case for informational advantages

Whether assessing the accuracy of expert forecasting, the pros and cons of group communication, or the value of evidence in diagnostic or predictive reasoning, dependencies between experts, group members, or evidence have traditionally been seen as a form of redundancy. We demonstrate that this conception of dependence conflates the structure of a dependency network, and the observations across this network. By disentangling these two elements we show, via mathematical proof and specific examples, that there are cases where dependencies yield an informational advantage over independence. More precisely, when a structural dependency exists, but observations are either partial or contradicting, these observations provide more support to a hypothesis than when this structural dependency does not exist, ceteris paribus. Furthermore, we show that lay reasoners endorse sufficient assumptions underpinning these advantageous structures yet fail to appreciate their implications for probability judgments and belief revision.

Majority rules: how good are we at aggregating convergent opinions?

Mathematical models and simulations demonstrate the power of majority rules, i.e. following an opinion shared by a majority of group members. Majority opinion should be followed more when (a) the relative and absolute size of the majority grow, the members of the majority are (b) competent, and (c) benevolent, (d) the majority opinion conflicts less with our prior beliefs and (e) the members of the majority formed their opinions independently. We review the experimental literature bearing on these points. The few experiments bearing on (b) and (c) suggest that both factors are adequately taken into account. Many experiments show that (d) is also followed, with participants usually putting too much weight on their own opinion relative to that of the majority. Regarding factors (a) and (e), in contrast, the evidence is mixed: participants sometimes take into account optimally the absolute and relative size of the majority, as well as the presence of informational dependencies. In other circumstances, these factors are ignored. We suggest that an evolutionary framework can help make sense of these conflicting results by distinguishing between evolutionarily valid cues - that are readily taken into account - and non-evolutionarily valid cues - that are ignored by default.

The Benefit of Additional Opinions

In daily decision making, people often solicit one another's opinions in the hope of improving their own judgment. According to both theory and empirical results, integrating even a few opinions is beneficial, with the accuracy gains diminishing as the bias of the judges or the correlation between their opinions increases. Decision makers using intuitive policies for integrating others' opinions rely on a variety of accuracy cues in weighting the opinions they receive. They tend to discount dissenters and to give greater weight to their own opinion than to other people's opinions.

But is it social? How to tell when groups are more than the sum of their members

Failure to distinguish between statistical effects and genuine social interaction may lead to unwarranted conclusions about the role of self-differentiation in group function. We offer an introduction to these issues from the perspective of recent research on collaborative cognition.

Public Reception of Climate Science: Coherence, Reliability, and Independence

Possible measures to mitigate climate change require global collective actions whose impacts will be felt by many, if not all. Implementing such actions requires successful communication of the reasons for them, and hence the underlying climate science, to a degree that far exceeds typical scientific issues which do not require large-scale societal response. Empirical studies have identified factors, such as the perceived level of consensus in scientific opinion and the perceived reliability of scientists, that can limit people's trust in science communicators and their subsequent acceptance of climate change claims. Little consideration has been given, however, to recent formal results within philosophy concerning the relationship between truth, the reliability of evidence sources, the coherence of multiple pieces of evidence/testimonies, and the impact of (non-)independence between sources of evidence. This study draws on these results to evaluate exactly what has (and, more important, has not yet) been established in the empirical literature about the factors that bias the public's reception of scientific communications about climate change.

Knowing the crowd within: Metacognitive limits on combining multiple judgments

We investigated how decision-makers use multiple opportunities to judge a quantity. Decision-makers undervalue the benefit of combining their own judgment with an advisor's, but theories disagree about whether this bias would apply to combining several of one's own judgments. Participants estimated percentage answers to general knowledge questions (e.g., What percent of the world's population uses the Internet?) on two occasions. In a final decision phase, they selected their first, second, or average estimate to report for each question. We manipulated the cues available for this final decision. Given cues to general theories (the labels first guess, second guess, average), participants mostly averaged, but no more frequently on trials where the average was most accurate. Given item-specific cues (numerical values of the options), metacognitive accuracy was at chance. Given both cues, participants mostly averaged and switched strategies based on whichever yielded the most accurate value on a given trial. These results indicate that underappreciation of averaging estimates does not stem only from social differences between the self and an advisor and that combining general and item-specific cues benefits metacognition.

Cognitive “Category-Based Induction” Research and Social “Persuasion” Research Are Each About What Makes Arguments Believable

Social and cognitive psychologists each study factors that influence the believability of arguments, but they have worked mostly in parallel. We briefly examine and compare the dominant theories explaining argument believability in the social persuasion literature and the cognitive category-based induction literature. Although the two areas ask similar questions, they use different paradigms to study different aspects of the issues. We describe each area’s major paradigms and questions and then examine the conclusions that each area draws regarding the role of five variables important to argument believability: (a) the number of sources/premises, (b) the similarity between sources/premises, (c) individual differences in characteristics of the reasoner, (d) the available resources, and (e) the reasoner’s background knowledge and beliefs. Comparing the two literatures provides a more complete picture of the factors influencing argument believability and provides fruitful new avenues for integration and exploration.

The wisdom of many in one mind: improving individual judgments with dialectical bootstrapping

The "wisdom of crowds" in making judgments about the future or other unknown events is well established. The average quantitative estimate of a group of individuals is consistently more accurate than the typical estimate, and is sometimes even the best estimate. Although individuals' estimates may be riddled with errors, averaging them boosts accuracy because both systematic and random errors tend to cancel out across individuals. We propose exploiting the power of averaging to improve estimates generated by a single person by using an approach we call dialectical bootstrapping. Specifically, it should be possible to reduce a person's error by averaging his or her first estimate with a second one that harks back to somewhat different knowledge. We derive conditions under which dialectical bootstrapping fosters accuracy and provide an empirical demonstration that its benefits go beyond reliability gains. A single mind can thus simulate the wisdom of many.

Eliciting expert opinion for economic models: an applied example

OBJECTIVES

Expert opinion is considered as a legitimate source of information for decision-analytic modeling where required data are unavailable. Our objective was to develop a practical computer-based tool for eliciting expert opinion about the shape of the uncertainty distribution around individual model parameters.

METHODS

We first developed a prepilot survey with departmental colleagues to test a number of alternative approaches to eliciting opinions on the shape of the uncertainty distribution around individual parameters. This information was used to develop a survey instrument for an applied clinical example. This involved eliciting opinions from experts to inform a number of parameters involving Bernoulli processes in an economic model evaluating DNA testing for families with a genetic disease, hypertrophic cardiomyopathy. The experts were cardiologists, clinical geneticists, and laboratory scientists working with cardiomyopathy patient populations and DNA testing.

RESULTS

Our initial prepilot work suggested that the more complex elicitation techniques advocated in the literature were difficult to use in practice. In contrast, our approach achieved a reasonable response rate (50%), provided logical answers, and was generally rated as easy to use by respondents. The computer software user interface permitted graphical feedback throughout the elicitation process. The distributions obtained were incorporated into the model, enabling the use of probabilistic sensitivity analysis.

CONCLUSION

There is clearly a gap in the literature between theoretical elicitation techniques and tools that can be used in applied decision-analytic models. The results of this methodological study are potentially valuable for other decision analysts deriving expert opinion.

Overconfidence in Interval Estimates

Judges were asked to make numerical estimates (e.g., "In what year was the first flight of a hot air balloon?"). Judges provided high and low estimates such that they were X% sure that the correct answer lay between them. They exhibited substantial overconfidence: The correct answer fell inside their intervals much less than X% of the time. This contrasts with choices between 2 possible answers to a question, which showed much less overconfidence. The authors show that overconfidence in interval estimates can result from variability in setting interval widths. However, the main cause is that subjective intervals are systematically too narrow given the accuracy of one's information-sometimes only 40% as large as necessary to be well calibrated. The degree of overconfidence varies greatly depending on how intervals are elicited. There are also substantial differences among domains and between male and female judges. The authors discuss the possible psychological mechanisms underlying this pattern of findings.

Confidence in aggregation of expert opinions

We investigate the case of a single decision maker (DM) who obtains probabilistic forecasts regarding the occurrence of a unique target event from J distinct, symmetric, and equally diagnostic expert advisors (judges). The paper begins with a mathematical model of DM's aggregation process of expert opinions, in which confidence in the final aggregate is shown to be inversely related to its perceived variance. As such, confidence is expected to vary as a function of factors such as the number of experts, the total number of cues, the fraction of cues available to each expert, the level of inter-expert overlap in information, and the range of experts' opinions. In the second part of the paper, we present results from two experiments that support the main (ordinal) predictions of the model.