The Wisdom of Individuals: Exploring People's Knowledge About Everyday Events Using Iterated Learning

Drawing attention to previous studies can reduce confidence in a new research finding, even when confidence should increase

People often learn of new scientific findings from brief news reports, and may discount or ignore prior research, potentially contributing to misunderstanding of findings. In this preregistered study, we investigated how people interpret a brief news report on a new drug for weight loss. Participants read an article that either highlighted the importance of prior research when judging the drug's effectiveness, or made no mention of this issue. For articles describing no prior research, mean confidence in the drug was 62%. For articles that noted prior research was conducted, confidence increased as the proportion of studies with positive findings increased. When prior research was highlighted, confidence decreased by a small amount, even when it should have increased (i.e., even when most of the evidence supported the drug's effectiveness). Thus, people's judgements were more sceptical, but not necessarily more accurate. Judgements were not affected by education level, statistics experience, or personal relevance of the research topic.

A cultural evolutionary theory that explains both gradual and punctuated change

Cumulative cultural evolution (CCE) occurs among humans who may be presented with many similar options from which to choose, as well as many social influences and diverse environments. It is unknown what general principles underlie the wide range of CCE dynamics and whether they can all be explained by the same unified paradigm. Here, we present a scalable evolutionary model of discrete choice with social learning, based on a few behavioural science assumptions. This paradigm connects the degree of transparency in social learning to the human tendency to imitate others. Computer simulations and quantitative analysis show the interaction of three primary factors-information transparency, popularity bias and population size-drives the pace of CCE. The model predicts a stable rate of evolutionary change for modest degrees of popularity bias. As popularity bias grows, the transition from gradual to punctuated change occurs, with maladaptive subpopulations arising on their own. When the popularity bias gets too severe, CCE stops. This provides a consistent framework for explaining the rich and complex adaptive dynamics taking place in the real world, such as modern digital media.

How experts' own inconsistency relates to their confidence and between-expert disagreement

People routinely rely on experts' advice to guide their decisions. However, experts are known to make inconsistent judgments when judging the same case twice. Previous research on expert inconsistency has largely focused on individual or situational factors; here we focus directly on the cases themselves. First, using a theoretical model, we study how within-expert inconsistency and confidence are related to how strongly experts agree on a case. Second, we empirically test the model's predictions in two real-world datasets with a diagnostic ground truth from follow-up research: diagnosticians rating the same mammograms or images of the lower spine twice. Our modeling and empirical analyses converge on the same novel results: The more experts disagree in their initial decisions about a case (i.e., as consensus decreases), the less confident individual experts are in their initial decision-despite not knowing the level of consensus-and the more likely they are to judge that same case differently when facing it again months later, regardless of whether the expert consensus is correct. Our results suggest the following advice when faced with two conflicting decisions from a single expert: In the absence of more predictive cues, choose the more confident decision.

Culture without copying or selection

Typical examples of cultural phenomena all exhibit a degree of similarity across time and space at the level of the population. As such, a fundamental question for any science of culture is, what ensures this stability in the first place? Here we focus on the evolutionary and stabilising role of 'convergent transformation', in which one item causes the production of another item whose form tends to deviate from the original in a directed, non-random way. We present a series of stochastic models of cultural evolution investigating its effects. The results show that cultural stability can emerge and be maintained by virtue of convergent transformation alone, in the absence of any form of copying or selection process. We show how high-fidelity copying and convergent transformation need not be opposing forces, and can jointly contribute to cultural stability. We finally analyse how non-random transformation and high-fidelity copying can have different evolutionary signatures at population level, and hence how their distinct effects can be distinguished in empirical records. Collectively, these results supplement existing approaches to cultural evolution based on the Darwinian analogy, while also providing formal support for other frameworks - such as Cultural Attraction Theory - that entail its further loosening.

Social media summary

Culture can be produced and maintained by convergent transformation, without copying or selection involved.

Accuracy-Risk Trade-Off Due to Social Learning in Crowd-Sourced Financial Predictions

A critical question relevant to the increasing importance of crowd-sourced-based finance is how to optimize collective information processing and decision-making. Here, we investigate an often under-studied aspect of the performance of online traders: beyond focusing on just accuracy, what gives rise to the trade-off between risk and accuracy at the collective level? Answers to this question will lead to designing and deploying more effective crowd-sourced financial platforms and to minimizing issues stemming from risk such as implied volatility. To investigate this trade-off, we conducted a large online Wisdom of the Crowd study where 2037 participants predicted the prices of real financial assets (S&P 500, WTI Oil and Gold prices). Using the data collected, we modeled the belief update process of participants using models inspired by Bayesian models of cognition. We show that subsets of predictions chosen based on their belief update strategies lie on a Pareto frontier between accuracy and risk, mediated by social learning. We also observe that social learning led to superior accuracy during one of our rounds that occurred during the high market uncertainty of the Brexit vote.

Coincidence judgment in causal reasoning: How coincidental is this?

Given the important conceptual connections between cause and coincidence as well as the extensive prior research on causality asking, "how causal is this?", the present research proposes and evaluated a psychological construction of coincidentality as the answer to the question, "how coincidental is this?" Four experiments measured the judgment properties of a reasonably large set of real coincidences from an initial diary study. These judgements included coincidentality and an array of other judgments about event uncertainty, hypothesis belief and surprise as predictors of coincidentality consistent with and supporting our prior definition of coincidence (Johansen & Osman, 2015): "coincidences are surprising pattern repetitions that are observed to be unlikely by chance but are nonetheless ascribed to chance since the search for causal mechanisms has not produced anything more plausible than mere chance." In particular, we evaluated formal models based on judgements of uncertainty, belief and surprise as predictors to develop a model of coincidentality. Ultimately, we argue that coincidentality is a marker for causal suspicion/discovery in terms of a flag that a new, unknown causal mechanism may be operating.

Why Higher Working Memory Capacity May Help You Learn: Sampling, Search, and Degrees of Approximation

Algorithms for approximate Bayesian inference, such as those based on sampling (i.e., Monte Carlo methods), provide a natural source of models of how people may deal with uncertainty with limited cognitive resources. Here, we consider the idea that individual differences in working memory capacity (WMC) may be usefully modeled in terms of the number of samples, or "particles," available to perform inference. To test this idea, we focus on two recent experiments that report positive associations between WMC and two distinct aspects of categorization performance: the ability to learn novel categories, and the ability to switch between different categorization strategies ("knowledge restructuring"). In favor of the idea of modeling WMC as a number of particles, we show that a single model can reproduce both experimental results by varying the number of particles-increasing the number of particles leads to both faster category learning and improved strategy-switching. Furthermore, when we fit the model to individual participants, we found a positive association between WMC and best-fit number of particles for strategy switching. However, no association between WMC and best-fit number of particles was found for category learning. These results are discussed in the context of the general challenge of disentangling the contributions of different potential sources of behavioral variability.

Category effects on stimulus estimation: Shifting and skewed frequency distributions-A reexamination

Duffy, Huttenlocher, Hedges, and Crawford (Psychonomic Bulletin & Review, 17(2), 224-230, 2010) report on experiments where participants estimate the lengths of lines. These studies were designed to test the category adjustment model (CAM), a Bayesian model of judgments. The authors report that their analysis provides evidence consistent with CAM: that there is a bias toward the running mean and not recent stimuli. We reexamine their data. First, we attempt to replicate their analysis, and we obtain different results. Second, we conduct a different statistical analysis. We find significant recency effects, and we identify several specifications where the running mean is not significantly related to judgment. Third, we conduct tests of auxiliary predictions of CAM. We do not find evidence that the bias toward the mean increases with exposure to the distribution. We also do not find that responses longer than the maximum of the distribution or shorter than the minimum become less likely with greater exposure to the distribution. Fourth, we produce a simulated dataset that is consistent with key features of CAM, and our methods correctly identify it as consistent with CAM. We conclude that the Duffy et al. (2010) dataset is not consistent with CAM. We also discuss how conventions in psychology do not sufficiently reduce the likelihood of these mistakes in future research. We hope that the methods that we employ will be used to evaluate other datasets.

Influence and seepage: An evidence-resistant minority can affect public opinion and scientific belief formation

Some well-established scientific findings may be rejected by vocal minorities because the evidence is in conflict with political views or economic interests. For example, the tobacco industry denied the medical consensus on the harms of smoking for decades, and the clear evidence about human-caused climate change is currently being rejected by many politicians and think tanks that oppose regulatory action. We present an agent-based model of the processes by which denial of climate change can occur, how opinions that run counter to the evidence can affect the scientific community, and how denial can alter the public discourse. The model involves an ensemble of Bayesian agents, representing the scientific community, that are presented with the emerging historical evidence of climate change and that also communicate the evidence to each other. Over time, the scientific community comes to agreement that the climate is changing. When a minority of agents is introduced that is resistant to the evidence, but that enter into the scientific discussion, the simulated scientific community still acquires firm knowledge but consensus formation is delayed. When both types of agents are communicating with the general public, the public remains ambivalent about the reality of climate change. The model captures essential aspects of the actual evolution of scientific and public opinion during the last 4 decades.

When Extremists Win: Cultural Transmission Via Iterated Learning When Populations Are Heterogeneous

How does the process of information transmission affect the cultural or linguistic products that emerge? This question is often studied experimentally and computationally via iterated learning, a procedure in which participants learn from previous participants in a chain. Iterated learning is a powerful tool because, when all participants share the same priors, the stationary distributions of the iterated learning chains reveal those priors. In many situations, however, it is unreasonable to assume that all participants share the same prior beliefs. We present four simulation studies and one experiment demonstrating that when the population of learners is heterogeneous, the behavior of an iterated learning chain can be unpredictable and is often systematically distorted by the learners with the most extreme biases. This results in group-level outcomes that reflect neither the behavior of any individuals within the population nor the overall population average. We discuss implications for the use of iterated learning as a methodological tool as well as for the processes that might have shaped cultural and linguistic evolution in the real world.

How effective is incidental learning of the shape of probability distributions?

The idea that people learn detailed probabilistic generative models of the environments they interact with is intuitively appealing, and has received support from recent studies of implicit knowledge acquired in daily life. The goal of this study was to see whether people efficiently induce a probability distribution based upon incidental exposure to an unknown generative process. Subjects played a 'whack-a-mole' game in which they attempted to click on objects appearing briefly, one at a time on the screen. Horizontal positions of the objects were generated from a bimodal distribution. After 180 plays of the game, subjects were unexpectedly asked to generate another 180 target positions of their own from the same distribution. Their responses did not even show a bimodal distribution, much less an accurate one (Experiment 1). The same was true for a pre-announced test (Experiment 2). On the other hand, a more extreme bimodality with zero density in a middle region did produce some distributional learning (Experiment 3), perhaps reflecting conscious hypothesis testing. We discuss the challenge this poses to the idea of efficient accurate distributional learning.

The anchor integration model: A descriptive model of anchoring effects

Few experimental effects in the psychology of judgment and decision making have been studied as meticulously as the anchoring effect. Although the existing literature provides considerable insight into the psychological processes underlying anchoring effects, extant theories up to this point have only generated qualitative predictions. While these theories have been productive in advancing our understanding of the underlying anchoring process, they leave much to be desired in the interpretation of specific anchoring effects. In this article, we introduce the Anchor Integration Model (AIM) as a descriptive tool for the measurement and quantification of anchoring effects. We develop two versions the model: one suitable for assessing between-participant anchoring effects, and another for assessing individual differences in anchoring effects. We then fit each model to data from two experiments, and demonstrate the model's utility in describing anchoring effects.

Rational Irrationality: Modeling Climate Change Belief Polarization Using Bayesian Networks

Belief polarization is said to occur when two people respond to the same evidence by updating their beliefs in opposite directions. This response is considered to be "irrational" because it involves contrary updating, a form of belief updating that appears to violate normatively optimal responding, as for example dictated by Bayes' theorem. In light of much evidence that people are capable of normatively optimal behavior, belief polarization presents a puzzling exception. We show that Bayesian networks, or Bayes nets, can simulate rational belief updating. When fit to experimental data, Bayes nets can help identify the factors that contribute to polarization. We present a study into belief updating concerning the reality of climate change in response to information about the scientific consensus on anthropogenic global warming (AGW). The study used representative samples of Australian and U.S.

PARTICIPANTS

Among Australians, consensus information partially neutralized the influence of worldview, with free-market supporters showing a greater increase in acceptance of human-caused global warming relative to free-market opponents. In contrast, while consensus information overall had a positive effect on perceived consensus among U.S. participants, there was a reduction in perceived consensus and acceptance of human-caused global warming for strong supporters of unregulated free markets. Fitting a Bayes net model to the data indicated that under a Bayesian framework, free-market support is a significant driver of beliefs about climate change and trust in climate scientists. Further, active distrust of climate scientists among a small number of U.S. conservatives drives contrary updating in response to consensus information among this particular group.

But is it social? How to tell when groups are more than the sum of their members

Failure to distinguish between statistical effects and genuine social interaction may lead to unwarranted conclusions about the role of self-differentiation in group function. We offer an introduction to these issues from the perspective of recent research on collaborative cognition.

Exploring the knowledge behind predictions in everyday cognition: an iterated learning study

Making accurate predictions about events is an important but difficult task. Recent work suggests that people are adept at this task, making predictions that reflect surprisingly accurate knowledge of the distributions of real quantities. Across three experiments, we used an iterated learning procedure to explore the basis of this knowledge: to what extent is domain experience critical to accurate predictions and how accurate are people when faced with unfamiliar domains? In Experiment 1, two groups of participants, one resident in Australia, the other in China, predicted the values of quantities familiar to both (movie run-times), unfamiliar to both (the lengths of Pharaoh reigns), and familiar to one but unfamiliar to the other (cake baking durations and the lengths of Beijing bus routes). While predictions from both groups were reasonably accurate overall, predictions were inaccurate in the selectively unfamiliar domains and, surprisingly, predictions by the China-resident group were also inaccurate for a highly familiar domain: local bus route lengths. Focusing on bus routes, two follow-up experiments with Australia-resident groups clarified the knowledge and strategies that people draw upon, plus important determinants of accurate predictions. For unfamiliar domains, people appear to rely on extrapolating from (not simply directly applying) related knowledge. However, we show that people's predictions are subject to two sources of error: in the estimation of quantities in a familiar domain and extension to plausible values in an unfamiliar domain. We propose that the key to successful predictions is not simply domain experience itself, but explicit experience of relevant quantities.

Decision from Models: Generalizing Probability Information to Novel Tasks

We investigate a new type of decision under risk where-to succeed-participants must generalize their experience in one set of tasks to a novel set of tasks. We asked participants to trade distance for reward in a virtual minefield where each successive step incurred the same fixed probability of failure (referred to as hazard). With constant hazard, the probability of success (the survival function) decreases exponentially with path length. On each trial, participants chose between a shorter path with smaller reward and a longer (more dangerous) path with larger reward. They received feedback in 160 training trials: encountering a mine along their chosen path resulted in zero reward and successful completion of the path led to the reward associated with the path chosen. They then completed 600 no-feedback test trials with novel combinations of path length and rewards. To maximize expected gain, participants had to learn the correct exponential model in training and generalize it to the test conditions. We compared how participants discounted reward with increasing path length to the predictions of nine choice models including the correct exponential model. The choices of a majority of the participants were best accounted for by a model of the correct exponential form although with marked overestimation of the hazard rate. The decision-from-models paradigm differs from experience-based decision paradigms such as decision-from-sampling in the importance assigned to generalizing experience-based information to novel tasks. The task itself is representative of everyday tasks involving repeated decisions in stochastically invariant environments.

Manifesto for a new (computational) cognitive revolution

The cognitive revolution offered an alternative to merely analyzing human behavior, using the notion of computation to rigorously express hypotheses about the mind. Computation also gives us new tools for testing these hypotheses - large behavioral databases generated by human interactions with computers and with one another. This kind of data is typically analyzed by computer scientists, who focus on predicting people's behavior based on their history. A new cognitive revolution is needed, demonstrating the value of minds as intervening variables in these analyses and using the results to evaluate models of human cognition.

Do Student Evaluations of University Reflect Inaccurate Beliefs or Actual Experience? A Relative Rank Model

It was shown that student satisfaction ratings are influenced by context in ways that have important theoretical and practical implications. Using questions from the UK's National Student Survey, the study examined whether and how students' expressed satisfaction with issues such as feedback promptness and instructor enthusiasm depends on the context of comparison (such as possibly inaccurate beliefs about the feedback promptness or enthusiasm experienced at other universities) that is evoked. Experiment 1 found strong effects of experimentally provided comparison context—for example, satisfaction with a given feedback time depended on the time's relative position within a context. Experiment 2 used a novel distribution-elicitation methodology to determine the prior beliefs of individual students about what happens in universities other than their own. It found that these beliefs vary widely and that students' satisfaction was predicted by how they believed their experience ranked within the distribution of others' experiences. A third study found that relative judgement principles also predicted students' intention to complain. An extended model was developed to show that purely rank-based principles of judgement can account for findings previously attributed to range effects. It was concluded that satisfaction ratings and quality of provision are different quantities, particularly when the implicit context of comparison includes beliefs about provision at other universities. Quality and satisfaction should be assessed separately, with objective measures (such as actual times to feedback), rather than subjective ratings (such as satisfaction with feedback promptness), being used to measure quality wherever practicable. © 2014 The Authors. Journal of Behavioral Decision Making published by John Wiley & Sons Ltd.

One and done? Optimal decisions from very few samples

In many learning or inference tasks human behavior approximates that of a Bayesian ideal observer, suggesting that, at some level, cognition can be described as Bayesian inference. However, a number of findings have highlighted an intriguing mismatch between human behavior and standard assumptions about optimality: People often appear to make decisions based on just one or a few samples from the appropriate posterior probability distribution, rather than using the full distribution. Although sampling-based approximations are a common way to implement Bayesian inference, the very limited numbers of samples often used by humans seem insufficient to approximate the required probability distributions very accurately. Here, we consider this discrepancy in the broader framework of statistical decision theory, and ask: If people are making decisions based on samples--but as samples are costly--how many samples should people use to optimize their total expected or worst-case reward over a large number of decisions? We find that under reasonable assumptions about the time costs of sampling, making many quick but locally suboptimal decisions based on very few samples may be the globally optimal strategy over long periods. These results help to reconcile a large body of work showing sampling-based or probability matching behavior with the hypothesis that human cognition can be understood in Bayesian terms, and they suggest promising future directions for studies of resource-constrained cognition.

Sentencing, severity, and social norms: a rank-based model of contextual influence on judgments of crimes and punishments

Context effects have been shown to bias lay people's evaluations of the severity of crimes and punishments. To investigate the cognitive mechanisms behind these effects, we develop and apply a rank-based social norms approach to judgments of perceived crime seriousness and sentence appropriateness. In Study 1, we find that (a) people believe on average that 84% of people illegally download software more than they do themselves and (b) their judged severity of, and concern about, their own illegal software downloading is predicted not by its amount but by how this amount is believed (typically inaccurately) to rank within a social comparison distribution. Studies 2 and 3 find that the judged appropriateness of a given sentence length is highly dependent on the length of other sentences available in the decision-making context: The same objective sentence was judged as approximately four times stricter when it was the second longest sentence being considered than when it was the fifth longest. It is concluded that the same mechanisms that are used to judge the magnitude of psychophysical stimuli bias judgments about legal matters.

Bayesian computation and mechanism: Theoretical pluralism drives scientific emergence

The breadth-first search adopted by Bayesian researchers to map out the conceptual space and identify what the framework can do is beneficial for science and reflective of its collaborative and incremental nature. Theoretical pluralism among researchers facilitates refinement of models within various levels of analysis, which ultimately enables effective cross-talk between different levels of analysis.

Exemplar models as a mechanism for performing Bayesian inference

Probabilistic models have recently received much attention as accounts of human cognition. However, most research in which probabilistic models have been used has been focused on formulating the abstract problems behind cognitive tasks and their optimal solutions, rather than on mechanisms that could implement these solutions. Exemplar models are a successful class of psychological process models in which an inventory of stored examples is used to solve problems such as identification, categorization, and function learning. We show that exemplar models can be used to perform a sophisticated form of Monte Carlo approximation known as importance sampling and thus provide a way to perform approximate Bayesian inference. Simulations of Bayesian inference in speech perception, generalization along a single dimension, making predictions about everyday events, concept learning, and reconstruction from memory show that exemplar models can often account for human performance with only a few exemplars, for both simple and relatively complex prior distributions. These results suggest that exemplar models provide a possible mechanism for implementing at least some forms of Bayesian inference.