The modulation of savouring by prediction error and its effects on choice

The causal structure and computational value of narratives

Many human behavioral and brain imaging studies have used narratively structured stimuli (e.g., written, audio, or audiovisual stories) to better emulate real-world experience in the laboratory. However, narratives are a special class of real-world experience, largely defined by their causal connections across time. Much contemporary neuroscience research does not consider this key property. We review behavioral and neuroscientific work that speaks to how causal structure shapes comprehension of and memory for narratives. We further draw connections between this work and reinforcement learning, highlighting how narratives help link causes to outcomes in complex environments. By incorporating the plausibility of causal connections between classes of actions and outcomes, reinforcement learning models may become more ecologically valid, while simultaneously elucidating the value of narratives.

Explaining dopamine through prediction errors and beyond

The most influential account of phasic dopamine holds that it reports reward prediction errors (RPEs). The RPE-based interpretation of dopamine signaling is, in its original form, probably too simple and fails to explain all the properties of phasic dopamine observed in behaving animals. This Perspective helps to resolve some of the conflicting interpretations of dopamine that currently exist in the literature. We focus on the following three empirical challenges to the RPE theory of dopamine: why does dopamine (1) ramp up as animals approach rewards, (2) respond to sensory and motor features and (3) influence action selection? We argue that the prediction error concept, once it has been suitably modified and generalized based on an analysis of each computational problem, answers each challenge. Nonetheless, there are a number of additional empirical findings that appear to demand fundamentally different theoretical explanations beyond encoding RPE. Therefore, looking forward, we discuss the prospects for a unifying theory that respects the diversity of dopamine signaling and function as well as the complex circuitry that both underlies and responds to dopaminergic transmission.

The effect of noninstrumental information on reward learning

Investigations of information-seeking often highlight people's tendency to forgo financial reward in return for advance information about future outcomes. Most of these experiments use tasks in which reward contingencies are described to participants. The use of such descriptions leaves open the question of whether the opportunity to obtain such noninstrumental information influences people's ability to learn and represent the underlying reward structure of an experimental environment. In two experiments, participants completed a two-armed bandit task with monetary incentives where reward contingencies were learned via trial-by-trial experience. We find, akin to description-based tasks, that participants are willing to forgo financial reward to receive information about a delayed, unchangeable outcome. Crucially, however, there is little evidence this willingness to pay for information is driven by an inaccurate representation of the reward structure: participants' representations approximated the underlying reward structure regardless of the presence of advance noninstrumental information. The results extend previous conclusions regarding the intrinsic value of information to an experience-based domain and highlight challenges of probing participants' memories for experienced rewards.

Temporal context effects on suboptimal choice

Choice can be driven both by rewards and stimuli that signal those rewards. Under certain conditions, pigeons will prefer options that lead to less probable reward when the reward is signaled. A recently quantified model, the Signal for Good News (SiGN) model, assumes that in the context of uncertainty, signals for a reduced delay to reward reinforce choice. The SiGN model provides an excellent fit to previous results from pigeons and the current studies are the first to test a priori quantitative predictions. Pigeons chose between a suboptimal alternative that led to signaled 20% food and an optimal alternative that led to 50% food. The duration of the choice period was manipulated across conditions in two experiments. Pigeons strongly preferred the suboptimal alternative at the shorter durations and strongly preferred the optimal alternative at the longer durations. The results from both experiments fit well with predictions from the SiGN model and show that altering the duration of the choice period has a dramatic effect in that it changes which of the two options pigeons prefer. More generally, these results suggest that the relative value of options is not fixed, but instead depends on the temporal context.

Curiosity and the dynamics of optimal exploration

What drives our curiosity remains an elusive and hotly debated issue, with multiple hypotheses proposed but a cohesive account yet to be established. This review discusses traditional and emergent theories that frame curiosity as a desire to know and a drive to learn, respectively. We adopt a model-based approach that maps the temporal dynamics of various factors underlying curiosity-based exploration, such as uncertainty, information gain, and learning progress. In so doing, we identify the limitations of past theories and posit an integrated account that harnesses their strengths in describing curiosity as a tool for optimal environmental exploration. In our unified account, curiosity serves as a 'common currency' for exploration, which must be balanced with other drives such as safety and hunger to achieve efficient action.

Individual differences in processing speed and curiosity explain infant habituation and dishabituation performance

Habituation and dishabituation are the most prevalent measures of infant cognitive functioning, and they have reliably been shown to predict later cognitive outcomes. Yet, the exact mechanisms underlying infant habituation and dishabituation are still unclear. To investigate them, we tested 106 8-month-old infants on a classic habituation task and a novel visual learning task. We used a hierarchical Bayesian model to identify individual differences in sustained attention, learning performance, processing speed and curiosity from the visual learning task. These factors were then related to habituation and dishabituation. We found that habituation time was related to individual differences in processing speed, while dishabituation was related to curiosity, but only for infants who did not habituate. These results offer novel insights in the mechanisms underlying habituation and serve as proof of concept for hierarchical models as an effective tool to measure individual differences in infant cognitive functioning. RESEARCH HIGHLIGHTS: We used a hierarchical Bayesian model to measure individual differences in infants' processing speed, learning performance, sustained attention, and curiosity. Faster processing speed was related to shorter habituation time. High curiosity was related to stronger dishabituation responses, but only for infants who did not habituate.

A special role for anterior cingulate cortex, but not orbitofrontal cortex or basolateral amygdala, in choices involving information

Subjects are often willing to pay a cost for information. In a procedure that promotes paradoxical choices, animals choose between a richer option followed by a cue that is rewarded 50% of the time (No Info) vs. a leaner option followed by one of two cues that signal certain outcomes: one always rewarded (100%) and the other never rewarded, 0% (Info). Since decisions involve comparing the subjective value of options after integrating all their features, preference for information may rely on cortico-amygdalar circuitry. To test this, male and female rats were prepared with bilateral inhibitory Designer Receptors Exclusively Activated by Designer Drugs (DREADDs) in the anterior cingulate cortex, orbitofrontal cortex, basolateral amygdala, or null virus (control). We inhibited these regions after stable preference was acquired. We found that inhibition of the anterior cingulate cortex destabilized choice preference in female rats without affecting latency to choose or response rate to cues. A logistic regression fit revealed that previous choice predicted current choice in all conditions, however previously rewarded Info trials strongly predicted preference in all conditions except in female rats following anterior cingulate cortex inhibition. The results reveal a causal, sex-dependent role for the anterior cingulate cortex in decisions involving information.

A neural mechanism for conserved value computations integrating information and rewards

Behavioral and economic theory dictate that we decide between options based on their values. However, humans and animals eagerly seek information about uncertain future rewards, even when this does not provide any objective value. This implies that decisions are made by endowing information with subjective value and integrating it with the value of extrinsic rewards, but the mechanism is unknown. Here, we show that human and monkey value judgements obey strikingly conserved computational principles during multi-attribute decisions trading off information and extrinsic reward. We then identify a neural substrate in a highly conserved ancient structure, the lateral habenula (LHb). LHb neurons signal subjective value, integrating information's value with extrinsic rewards, and the LHb predicts and causally influences ongoing decisions. Neurons in key input areas to the LHb largely signal components of these computations, not integrated value signals. Thus, our data uncover neural mechanisms of conserved computations underlying decisions to seek information about the future.

Curiosity evolves as information unfolds

When people feel curious, they often seek information to resolve their curiosity. Reaching resolution, however, does not always occur in a single step but instead may follow the accumulation of information over time. Here, we investigated changes in curiosity over a dynamic information-gathering process and how these changes related to affective and cognitive states as well as behavior. Human participants performed an Evolving Line Drawing Task, during which they reported guesses about the drawings' identities and made choices about whether to keep watching. In Study 1, the timing of choices was predetermined and externally imposed, while in Study 2, participants had agency in the timing of guesses and choices. Using this dynamic paradigm, we found that even within a single information-gathering episode, curiosity evolved in concert with other emotional states and with confidence. In both studies, we showed that the relationship between curiosity and confidence depended on stimulus entropy (unique guesses across participants) and on guess accuracy. We demonstrated that curiosity is multifaceted and can be experienced as either positive or negative depending on the state of information gathering. Critically, even when given the choice to alleviate uncertainty immediately (i.e., view a spoiler), higher curiosity promoted continuing to engage in the information-gathering process. Collectively, we show that curiosity changes over information accumulation to drive engagement with external stimuli, rather than to shortcut the path to resolution, highlighting the value inherent in the process of discovery.

The non-unitary nature of information preference

Factors affecting information-seeking behaviour can be task-endogenous (e.g., probability of winning a gamble), or task-exogenous (e.g., personality trait measures). Various task-endogenous factors affecting non-instrumental information-seeking behaviour have been identified, but it is unclear how task-exogenous factors affect such behaviour, and if they interact with task-endogenous factors. In an online information seeking experiment (N = 279), we focus on the role that outcome probability, as a task-endogenous factor, has on information preferences. We find reliable preference for advance information on highly probable gains and low preference for highly probable losses. Comparisons with individual trait measures of information preference (e.g., intolerance of uncertainty scale, obsessive-compulsive inventory, information preferences scale) reveal minimal association between these task-exogenous factors with choice task performance. We also find minimal interaction between outcome probability and individual trait measures. Despite the choice task and trait measures purportedly tapping the same (or similar) construct, the absence of clear relationships ultimately suggests a multi-dimensional nature of information preference.

The availability of non-instrumental information increases risky decision-making

Contemporary models of decision-making under risk focus on estimating the final value of each alternative course of action. According to such frameworks, information that has no capacity to alter a future payoff (i.e., is "non-instrumental") should have little effect on one's preference for risk. Importantly, however, recent work has shown that information, despite being non-instrumental, may nevertheless exert a striking influence on behavior. Here, we tested whether the opportunity to passively observe the sequence of events following a decision could modulate risky behavior, even if that information could not possibly influence the final result. Across three experiments, 71 individuals chose to accept or reject gambles on a five-window slot machine. If a gamble was accepted, each window was sequentially revealed prior to the outcome being declared. Critically, we informed participants about which windows would subsequently provide veridical information about the gamble outcome, should that gamble be accepted. Our analyses revealed three key findings. First, the opportunity to observe the consequences of one's choice significantly increased the likelihood of gambling, despite that information being entirely non-instrumental. Second, this effect generalized across different stakes. Finally, choices were driven predominantly by the likelihood that information could result in an earlier resolution of uncertainty. These findings demonstrate the importance of anticipatory information to decision-making under risk. More broadly, we provide strong evidence for the utility of non-instrumental information, by demonstrating its capacity to modulate primary economic decisions that should be driven by more motivationally salient variables associated with risk and reward.

Inefficient prioritization of task-relevant attributes during instrumental information demand

In natural settings, people evaluate complex multi-attribute situations and decide which attribute to request information about. Little is known about how people make this selection and specifically, how they identify individual observations that best predict the value of a multi-attribute situation. Here show that, in a simple task of information demand, participants inefficiently query attributes that have high individual value but are relatively uninformative about a total payoff. This inefficiency is robust in two instrumental conditions in which gathering less informative observations leads to significantly lower rewards. Across individuals, variations in the sensitivity to informativeness is associated with personality metrics, showing negative associations with extraversion and thrill seeking and positive associations with stress tolerance and need for cognition. Thus, people select informative queries using sub-optimal strategies that are associated with personality traits and influence consequential choices.

Emerging Principles of Attention and Information Demand

I review recent literature on information demand and its implications for attention control. I argue that this literature motivates a view of attention as a mechanism that reduces uncertainty by selectively sampling sensory stimuli on the basis of expected information gain (EIG). I discuss emerging evidence on how individuals estimate the two quantities that determine EIG, prior uncertainty and stimulus diagnosticity (predictive accuracy). I also discuss the neural mechanisms that compute EIG and integrate it with rewards in frontoparietal, executive, and neuromodulatory circuits. I end by considering the implications of this framework for a broader understanding of the factors that assign relevance to sensory stimuli and the role of attention in decision making and other cognitive functions.

Visuospatial information foraging describes search behavior in learning latent environmental features

In the real world, making sequences of decisions to achieve goals often depends upon the ability to learn aspects of the environment that are not directly perceptible. Learning these so-called latent features requires seeking information about them. Prior efforts to study latent feature learning often used single decisions, used few features, and failed to distinguish between reward-seeking and information-seeking. To overcome this, we designed a task in which humans and monkeys made a series of choices to search for shapes hidden on a grid. On our task, the effects of reward and information outcomes from uncovering parts of shapes could be disentangled. Members of both species adeptly learned the shapes and preferred to select tiles expected to be informative earlier in trials than previously rewarding ones, searching a part of the grid until their outcomes dropped below the average information outcome-a pattern consistent with foraging behavior. In addition, how quickly humans learned the shapes was predicted by how well their choice sequences matched the foraging pattern, revealing an unexpected connection between foraging and learning. This adaptive search for information may underlie the ability in humans and monkeys to learn latent features to support goal-directed behavior in the long run.

Paradoxical choice and the reinforcing value of information

Signals that reduce uncertainty can be valuable because well-informed decision-makers can better align their preferences to opportunities. However, some birds and mammals display an appetite for informative signals that cannot be used to increase returns. We explore the role that reward-predictive stimuli have in fostering such preferences, aiming at distinguishing between two putative underlying mechanisms. The 'information hypothesis' proposes that reducing uncertainty is reinforcing per se, somewhat consistently with the concept of curiosity: a motivation to know in the absence of tractable extrinsic benefits. In contrast, the 'conditioned reinforcement hypothesis', an associative account, proposes asymmetries in secondarily acquired reinforcement: post-choice stimuli announcing forthcoming rewards (S⁺) reinforce responses more than stimuli signalling no rewards (S^-) inhibit responses. In three treatments, rats faced two equally profitable options delivering food probabilistically after a fixed delay. In the informative option (Info), food or no food was signalled immediately after choice, whereas in the non-informative option (NoInfo) outcomes were uncertain until the delay lapsed. Subjects preferred Info when (1) both outcomes were explicitly signalled by salient auditory cues, (2) only forthcoming food delivery was explicitly signalled, and (3) only the absence of forthcoming reward was explicitly signalled. Acquisition was slower in (3), when food was not explicitly signalled, showing that signals for positive outcomes have a greater influence on the development of preference than signals for negative ones. Our results are consistent with an elaborated conditioned reinforcement account, and with the conjecture that both uncertainty reduction and conditioned reinforcement jointly act to generate preference.

People adaptively use information to improve their internal states and external outcomes

Information can strongly impact people's affect, their level of uncertainty and their decisions. It is assumed that people seek information with the goal of improving all three. But are they successful at achieving this goal? Answering this question is important for assessing the impact of self-driven information consumption on people's well-being. Here, over five experiments (total N = 727) we show that participants accurately predict the impact of information on their internal states (e.g., affect and cognition) and external outcomes (e.g., material rewards), and use these predictions to guide information-seeking choices. A model incorporating participants' subjective expectations regarding the impact of information on their affective, cognitive, and material outcomes accounted for information-seeking choices better than a model that included only objective proxies of those measures. This model also accounted for individual differences in information-seeking choices. By balancing considerations of the impact of information on affective, cognitive and material outcomes when seeking knowledge, participants became happier, more certain and made better decisions when they sought information relative to when they did not, suggesting that the actual consequences of receiving information aligned with their subjective expectations.

Uncertainty modulates visual maps during noninstrumental information demand

Animals are intrinsically motivated to obtain information independently of instrumental incentives. This motivation depends on two factors: a desire to resolve uncertainty by gathering accurate information and a desire to obtain positively-valenced observations, which predict favorable rather than unfavorable outcomes. To understand the neural mechanisms, we recorded parietal cortical activity implicated in prioritizing stimuli for spatial attention and gaze, in a task in which monkeys were free (but not trained) to obtain information about probabilistic non-contingent rewards. We show that valence and uncertainty independently modulated parietal neuronal activity, and uncertainty but not reward-related enhancement consistently correlated with behavioral sensitivity. The findings suggest uncertainty-driven and valence-driven information demand depend on partially distinct pathways, with the former being consistently related to parietal responses and the latter depending on additional mechanisms implemented in downstream structures.

Curiosity is motivated by uncertainty and valence, but how uncertainty and valence are encoded in the brain remains poorly understood. Here, the authors show that parietal neurons are enhanced by both factors, but that they specifically predict visual information seeking based on uncertainty.

A Model of Trust

Trust is central to a large variety of social interactions. Different research fields have empirically and theoretically investigated trust, observing trusting behaviors in different situations and pinpointing their different components and constituents. However, a unifying, computational formalization of those diverse components and constituents of trust is still lacking. Previous work has mainly used computational models borrowed from other fields and developed for other purposes to explain trusting behaviors in empirical paradigms. Here, I computationally formalize verbal models of trust in a simple model (i.e., vulnerability model) that combines current and prospective action values with beliefs and expectancies about a partner’s behavior. By using the classic investment game (IG)—an economic game thought to capture some important features of trusting behaviors in social interactions—I show how variations of a single parameter of the vulnerability model generates behaviors that can be interpreted as different “trust attitudes”. I then show how these behavioral patterns change as a function of an individual’s loss aversion and expectations of the partner’s behavior. I finally show how the vulnerability model can be easily extended in a novel IG paradigm to investigate inferences on different traits of a partner. In particular, I will focus on benevolence and competence—two character traits that have previously been described as determinants of trustworthiness impressions central to trust. The vulnerability model can be employed as is or as a utility function within more complex Bayesian frameworks to fit participants’ behavior in different social environments where actions are associated with subjective values and weighted by individual beliefs about others’ behaviors. Hence, the vulnerability model provides an important building block for future theoretical and empirical work across a variety of research fields.

Neurocomputational mechanisms underlying the subjective value of information

Humans have a striking desire to actively seek new information, even when it is devoid of any instrumental utility. However, the mechanisms that drive individuals' subjective preference for information remain unclear. Here, we used fMRI to examine the processing of subjective information value, by having participants decide how much effort they were willing to trade-off for non-instrumental information. We showed that choices were best described by a model that accounted for: (1) the variability in individuals' estimates of uncertainty, (2) their desire to reduce that uncertainty, and (3) their subjective preference for positively valenced information. Model-based analyses revealed the anterior cingulate as a key node that encodes the subjective value of information across multiple stages of decision-making - including when information was prospectively valued, and when the outcome was definitively delivered. These findings emphasise the multidimensionality of information value, and reveal the neurocomputational mechanisms underlying the variability in individuals' desire to physically pursue informative outcomes.

Curiosity or savouring? Information seeking is modulated by both uncertainty and valence

Curiosity is pervasive in our everyday lives, but we know little about the factors that contribute to this drive. In the current study, we assessed whether curiosity about uncertain outcomes is modulated by the valence of the information, i.e. whether the information is good or bad news. Using a lottery task in which outcome uncertainty, expected value and outcome valence (gain versus loss) were manipulated independently, we found that curiosity is overall higher for gains compared with losses and that curiosity increased with increasing outcome uncertainty for both gains and losses. These effects of uncertainty and valence did not interact, indicating that the motivation to reduce uncertainty and the motivation to maximize positive information represent separate, independent drives.

Efficiency and prioritization of inference-based credit assignment

Organisms adapt to their environments by learning to approach states that predict rewards and avoid states associated with punishments. Knowledge about the affective value of states often relies on credit assignment (CA), whereby state values are updated on the basis of reward feedback. Remarkably, humans assign credit to states that are not observed but are instead inferred based on a cognitive map that represents structural knowledge of an environment. A pertinent example is authors attempting to infer the identity of anonymous reviewers to assign them credit or blame and, on this basis, inform future referee recommendations. Although inference is cognitively costly, it is unknown how it influences CA or how it is apportioned between hidden and observable states (for example, both anonymous and revealed reviewers). We addressed these questions in a task that provided choices between lotteries where each led to a unique pair of occasionally rewarding outcome states. On some trials, both states were observable (rendering inference nugatory), whereas on others, the identity of one of the states was concealed. Importantly, by exploiting knowledge of choice-state associations, subjects could infer the identity of this hidden state. We show that having to perform inference reduces state-value updates. Strikingly, and in violation of normative theories, this reduction in CA was selective for the observed outcome alone. These findings have implications for the operation of putative cognitive maps.

Choosing increases the value of non-instrumental information

Curiosity pervades all aspects of human behaviour and decision-making. Recent research indicates that the value of information is determined by its propensity to reduce uncertainty, and the hedonic value of the outcomes it predicts. Previous findings also indicate a preference for options that are freely chosen, compared to equivalently valued alternatives that are externally assigned. Here, we asked whether the value of information also varies as a function of self- or externally-imposed choices. Participants rated their preference for information that followed either a self-chosen decision, or an externally imposed condition. Our results showed that choosing a lottery significantly increased the subjective value of information about the outcome. Computational modelling indicated that this change in information-seeking behaviour was not due to changes in the subjective probability of winning, but instead reflected an independent effect of choosing on the value of resolving uncertainty. These results demonstrate that agency over a prospect is an important source of information value.

The orbitofrontal cortex in temporal cognition

One of the most important factors in decision-making is estimating the value of available options. Subregions of the prefrontal cortex, including the orbitofrontal cortex (OFC), have been deemed essential for this process. Value computations require a complex integration across numerous dimensions, including, reward magnitude, effort, internal state, and time. The importance of the temporal dimension is well illustrated by temporal discounting tasks, in which subjects select between smaller-sooner versus larger-later rewards. The specific role of OFC in telling time and integrating temporal information into decision-making remains unclear. Based on the current literature, in this review we reevaluate current theories of OFC function, accounting for the influence of time. Incorporating temporal information into value estimation and decision-making requires distinct, yet interrelated, forms of temporal information including the ability to tell time, represent time, create temporal expectations, and the ability to use this information for optimal decision-making in a wide range of tasks, including temporal discounting and wagering. We use the term "temporal cognition" to refer to the integrated use of these different aspects of temporal information. We suggest that the OFC may be a critical site for the integration of reward magnitude and delay, and thus important for temporal cognition. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

People as Intuitive Scientists: Reconsidering Statistical Explanations of Decision Making

A persistent metaphor in decision-making research casts people as intuitive statisticians. Popular explanations based on this metaphor assume that the way in which people represent the environment is specified and fixed a priori. A major flaw in this account is that it is not clear how people know what aspects of an environment are important, how to interpret those aspects, and how to make decisions based on them. We suggest a theoretical reorientation away from assuming people's representations towards a focus on explaining how people themselves specify what is important to represent. This perspective casts decision makers as intuitive scientists able to flexibly construct, modify, and replace the representations of the decision problems they face.

Neural circuitry of information seeking

Humans and animals navigate uncertain environments by seeking information about the future. Remarkably, we often seek information even when it has no instrumental value for aiding our decisions - as if the information is a source of value in its own right. In recent years, there has been a flourishing of research into these non-instrumental information preferences and their implementation in the brain. Individuals value information about uncertain future rewards, and do so for multiple reasons, including valuing resolution of uncertainty and overweighting desirable information. The brain motivates this information seeking by tapping into some of the same circuitry as primary rewards like food and water. However, it also employs cortex and basal ganglia circuitry that predicts and values information as distinct from primary reward. Uncovering how these circuits cooperate will be fundamental to understanding information seeking and motivated behavior as a whole, in our increasingly complex and information-rich world.

The ventral striatum dissociates information expectation, reward anticipation, and reward receipt

Do dopaminergic reward structures represent the expected utility of information similarly to a reward? Optimal experimental design models from Bayesian decision theory and statistics have proposed a theoretical framework for quantifying the expected value of information that might result from a query. In particular, this formulation quantifies the value of information before the answer to that query is known, in situations where payoffs are unknown and the goal is purely epistemic: That is, to increase knowledge about the state of the world. Whether and how such a theoretical quantity is represented in the brain is unknown. Here we use an event-related functional MRI (fMRI) task design to disentangle information expectation, information revelation and categorization outcome anticipation, and response-contingent reward processing in a visual probabilistic categorization task. We identify a neural signature corresponding to the expectation of information, involving the left lateral ventral striatum. Moreover, we show a temporal dissociation in the activation of different reward-related regions, including the nucleus accumbens, medial prefrontal cortex, and orbitofrontal cortex, during information expectation versus reward-related processing.

Suboptimal choice by pigeons is eliminated when key-pecking behavior is replaced by treadle-pressing

In the study of suboptimal choice, a reliable result is that pigeons strongly prefer an alternative that signals whether a reinforcer will be delivered or not over another alternative without that information even if the first provides a lower probability of reinforcement. In the aforementioned research, key pecking has been the operant response and illuminated keys the discriminative stimuli. In the present study we modified both of these aspects of the procedure in order to analyze the generality of suboptimal preferences of pigeons and to investigate the effect of changes in the incentive salience of the discriminative stimuli. To accomplish this, we presented pigeons a choice situation with the same parameters of reinforcement than previous research, but with treadle pressing as the choice response and ambient lights as discriminative stimuli. Under these conditions, most of the pigeons showed optimal behavior and a high degree of discrimination of the stimuli associated with the discriminative alternative. A control condition with key pecking as choice response and keylights as discriminative stimuli showed that the same pigeons turned to be suboptimal, a result that discards the possibility that the optimality found in the main condition was a consequence of a particular characteristic of our sample of subjects or of our procedure. We discuss the influence that the attribution of incentive salience to the discriminative stimuli has on suboptimal choice in both pigeons and rats.

The value of what's to come: Neural mechanisms coupling prediction error and the utility of anticipation

Having something to look forward to is a keystone of well-being. Anticipation of future reward, such as an upcoming vacation, can often be more gratifying than the experience itself. Theories suggest the utility of anticipation underpins various behaviors, ranging from beneficial information-seeking to harmful addiction. However, how neural systems compute anticipatory utility remains unclear. We analyzed the brain activity of human participants as they performed a task involving choosing whether to receive information predictive of future pleasant outcomes. Using a computational model, we show three brain regions orchestrate anticipatory utility. Specifically, ventromedial prefrontal cortex tracks the value of anticipatory utility, dopaminergic midbrain correlates with information that enhances anticipation, while sustained hippocampal activity mediates a functional coupling between these regions. Our findings suggest a previously unidentified neural underpinning for anticipation's influence over decision-making and unify a range of phenomena associated with risk and time-delay preference.

A Neuro-computational Account of Arbitration between Choice Imitation and Goal Emulation during Human Observational Learning

When individuals learn from observing the behavior of others, they deploy at least two distinct strategies. Choice imitation involves repeating other agents' previous actions, whereas emulation proceeds from inferring their goals and intentions. Despite the prevalence of observational learning in humans and other social animals, a fundamental question remains unaddressed: how does the brain decide which strategy to use in a given situation? In two fMRI studies (the second a pre-registered replication of the first), we identify a neuro-computational mechanism underlying arbitration between choice imitation and goal emulation. Computational modeling, combined with a behavioral task that dissociated the two strategies, revealed that control over behavior was adaptively and dynamically weighted toward the most reliable strategy. Emulation reliability, the model's arbitration signal, was represented in the ventrolateral prefrontal cortex, temporoparietal junction, and rostral cingulate cortex. Our replicated findings illuminate the computations by which the brain decides to imitate or emulate others.

Autistic traits influence the strategic diversity of information sampling: Insights from two-stage decision models

Information sampling can reduce uncertainty in future decisions but is often costly. To maximize reward, people need to balance sampling cost and information gain. Here we aimed to understand how autistic traits influence the optimality of information sampling and to identify the particularly affected cognitive processes. Healthy human adults with different levels of autistic traits performed a probabilistic inference task, where they could sequentially sample information to increase their likelihood of correct inference and may choose to stop at any moment. We manipulated the cost and evidence associated with each sample and compared participants’ performance to strategies that maximize expected gain. We found that participants were overall close to optimal but also showed autistic-trait-related differences. Participants with higher autistic traits had a higher efficiency of winning rewards when the sampling cost was zero but a lower efficiency when the cost was high and the evidence was more ambiguous. Computational modeling of participants’ sampling choices and decision times revealed a two-stage decision process, with the second stage being an optional second thought. Participants may consider cost in the first stage and evidence in the second stage, or in the reverse order. The probability of choosing to stop sampling at a specific stage increases with increasing cost or increasing evidence. Surprisingly, autistic traits did not influence the decision in either stage. However, participants with higher autistic traits inclined to consider cost first, while those with lower autistic traits considered cost or evidence first in a more balanced way. This would lead to the observed autistic-trait-related advantages or disadvantages in sampling optimality, depending on whether the optimal sampling strategy is determined only by cost or jointly by cost and evidence.

Children with autism can spend hours practicing lining up toys or learning all about cars or lighthouses. This kind of behaviors, we think, may reflect suboptimal information sampling strategies, that is, a failure to balance the gain of information with the cost (time, energy, or money) of information sampling. We hypothesized that suboptimal information sampling is a general characteristic of people with autism or high level of autistic traits. In our experiment, we tested how participants may adjust their sampling strategies with the change of sampling cost and information gain in the environment. Though all participants were healthy young adults who had similar IQs, higher autistic traits were associated with higher or lower efficiency of winning rewards under different conditions. Counterintuitively, participants with different levels of autistic traits did not differ in the general tendency of oversampling or undersampling, or in the decision they would reach when a specific set of sampling cost or information gain was considered. Instead, participants with higher autistic traits consistently considered sampling cost first and only weighed information gain during a second thought, while those with lower autistic traits had more diverse sampling strategies that consequently better balanced sampling cost and information gain.

Contributions of anterior cingulate cortex and basolateral amygdala to decision confidence and learning under uncertainty

The subjective sense of certainty, or confidence, in ambiguous sensory cues can alter the interpretation of reward feedback and facilitate learning. We trained rats to report the orientation of ambiguous visual stimuli according to a spatial stimulus-response rule that must be learned. Following choice, rats could wait a self-timed delay for reward or initiate a new trial. Waiting times increase with discrimination accuracy, demonstrating that this measure can be used as a proxy for confidence. Chemogenetic silencing of BLA shortens waiting times overall whereas ACC inhibition renders waiting times insensitive to confidence-modulating attributes of visual stimuli, suggesting contribution of ACC but not BLA to confidence computations. Subsequent reversal learning is enhanced by confidence. Both ACC and BLA inhibition block this enhancement but via differential adjustments in learning strategies and consistent use of learned rules. Altogether, we demonstrate dissociable roles for ACC and BLA in transmitting confidence and learning under uncertainty.

Diverse motives for human curiosity

Curiosity-our desire to know-is a fundamental drive in human behaviour, but its mechanisms are poorly understood. A classical question concerns the curiosity motives. What drives individuals to become curious about some but not other sources of information?¹ Here we show that curiosity about probabilistic events depends on multiple aspects of the distribution of these events. Participants (n = 257) performed a task in which they could demand advance information about only one of two randomly selected monetary prizes that contributed to their income. Individuals differed markedly in the extent to which they requested information as a function of the ex ante uncertainty or ex ante value of an individual prize. This heterogeneity was not captured by theoretical models describing curiosity as a desire to learn about the total rewards of a situation^2,3. Instead, it could be explained by an extended model that allowed for attribute-specific anticipatory utility-the savouring of individual components of the eventual reward-and postulates that this utility increased nonlinearly with the certainty of receiving the reward. Parameter values fitting individual choices were consistent for information about gains or losses, suggesting that attribute-specific anticipatory utility captures fundamental heterogeneity in the determinants of curiosity.

Retrospective model-based inference guides model-free credit assignment

An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an interaction between model-free (MF) and model-based (MB) control systems. We present and support experimentally a theory of MB retrospective-inference. Within this framework, a MB system resolves uncertainty that prevailed when actions were taken thus guiding an MF credit-assignment. Using a task in which there was initial uncertainty about the lotteries that were chosen, we found that when participants' momentary uncertainty about which lottery had generated an outcome was resolved by provision of subsequent information, participants preferentially assigned credit within a MF system to the lottery they retrospectively inferred was responsible for this outcome. These findings extend our knowledge about the range of MB functions and the scope of system interactions.

Costly curiosity: People pay a price to resolve an uncertain gamble early

Humans are inherently curious creatures, continuously seeking out information about future outcomes. Such advance information is often valuable, potentially allowing people to select better courses of action. In non-human animals, this drive for information can be so strong that they forego food or water to find out a few seconds earlier whether an uncertain option will provide a reward. Here, we assess whether people will exhibit a similar sub-optimal preference for advance information. Participants played a card-flipping task where they were probabilistically rewarded based on the pattern of 3 cards that were revealed after a 5-s delay. During this delay, participants could instead pay a cost to find out the next card's identity immediately. This choice to find out early did not influence the eventual outcome. Participants preferred to find out early about 80% of the time when the information was free; they were even willing to incur an expense to get advance information about the eventual outcome. The expected magnitude of the outcome, however, had little impact on the likelihood of finding out early. These results suggest that humans, like animals, value non-instrumental information and will pay a price for such information, independent of its utility.

Computational mechanisms of curiosity and goal-directed exploration

Successful behaviour depends on the right balance between maximising reward and soliciting information about the world. Here, we show how different types of information-gain emerge when casting behaviour as surprise minimisation. We present two distinct mechanisms for goal-directed exploration that express separable profiles of active sampling to reduce uncertainty. 'Hidden state' exploration motivates agents to sample unambiguous observations to accurately infer the (hidden) state of the world. Conversely, 'model parameter' exploration, compels agents to sample outcomes associated with high uncertainty, if they are informative for their representation of the task structure. We illustrate the emergence of these types of information-gain, termed active inference and active learning, and show how these forms of exploration induce distinct patterns of 'Bayes-optimal' behaviour. Our findings provide a computational framework for understanding how distinct levels of uncertainty systematically affect the exploration-exploitation trade-off in decision-making.

Active Confirmation Bias in the Evaluative Processing of Food Images

Predictive processing is fundamental to many aspects of the human mind, including perception and decision-making. It remains to be elucidated, however, in which way predictive information impacts on evaluative processing, particularly in tasks that employ bivalent stimulus sets. Various accounts, including framing, proactive interference, and cognitive control, appear to imply contradictory proposals on the relation between prediction and preference formation. To disambiguate whether predictive cues produce congruent biases versus opponent mechanisms in evaluative processing, we conducted two experiments in which participants were asked to rate individual food images. The image database included appetitive and aversive items. In each trial, a cue predicted, with varying degrees of reliability, the valence of the impending food image. In both experiments, we found that the ratings exhibited congruent biases as a function of the reliability of the predictive cue, with the highest evaluations following the most reliable positive-valence predictions. Eye prepositioning further showed a selective spatial bias suggestive of response preparation in line with the predictions. The response times also exhibited a pattern of results consistent with selective preparation, producing slow responses following invalid predictions. The data suggested an active form of evaluative processing, implementing a confirmation bias that aims to accommodate the prediction.

Suboptimal choice in pigeons: Does the predictive value of the conditioned reinforcer alone determine choice?

Prior research has found that pigeons are indifferent between an option that always provides a signal for reinforcement and an alternative that provides a signal for reinforcement only 50% of the time (and a signal for the absence of reinforcement 50% of the time). This suboptimal choice suggests that the frequency of the signal for reinforcement plays virtually no role and choice depends only on the predictive value of the signal for reinforcement associated with each alternative. In the present research we tested the hypothesis that if there are two or three signals for reinforcement associated with the suboptimal alternative but each occurs only 25% or 17% of the time, respectively, pigeons would show a greater preference for the suboptimal alternative. Although we found that increasing the number of signals for reinforcement associated with the suboptimal alternative did not increase the preference for the suboptimal alternative (relative to a single signal for reinforcement) extended training on this task resulted in a significant preference for the suboptimal alternative by both groups. This result suggests that contrast between the expected outcome at the time of choice (50% reinforcement) and the value of the signal for reinforcement (100% reinforcement) is also responsible for choice of the suboptimal alternative.

Valuation of knowledge and ignorance in mesolimbic reward circuitry

Humans desire to know what the future holds. Yet, at times they decide to remain ignorant (e.g., reject medical screenings). These decisions have important societal implications in domains ranging from health to finance. We show how the opportunity to gain information is valued and explain why knowledge is not always preferred. Specifically, the mesolimbic reward circuitry selectively treats the opportunity to gain knowledge about favorable, but not unfavorable, outcomes as a reward to be approached. This coding predicts biased information seeking: Participants choose knowledge about future desirable outcomes more than about undesirable ones, vice versa for ignorance, and are willing to pay for both. This work demonstrates a role for valence in how the human brain values knowledge.

The pursuit of knowledge is a basic feature of human nature. However, in domains ranging from health to finance people sometimes choose to remain ignorant. Here, we show that valence is central to the process by which the human brain evaluates the opportunity to gain information, explaining why knowledge may not always be preferred. We reveal that the mesolimbic reward circuitry selectively treats the opportunity to gain knowledge about future favorable outcomes, but not unfavorable outcomes, as if it has positive utility. This neural coding predicts participants’ tendency to choose knowledge about future desirable outcomes more often than undesirable ones, and to choose ignorance about future undesirable outcomes more often than desirable ones. Strikingly, participants are willing to pay both for knowledge and ignorance as a function of the expected valence of knowledge. The orbitofrontal cortex (OFC), however, responds to the opportunity to receive knowledge over ignorance regardless of the valence of the information. Connectivity between the OFC and mesolimbic circuitry could contribute to a general preference for knowledge that is also modulated by valence. Our findings characterize the importance of valence in information seeking and its underlying neural computation. This mechanism could lead to suboptimal behavior, such as when people reject medical screenings or monitor investments more during bull than bear markets.

An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals

Serotonin has widespread, but computationally obscure, modulatory effects on learning and cognition. Here, we studied the impact of optogenetic stimulation of dorsal raphe serotonin neurons in mice performing a non-stationary, reward-driven decision-making task. Animals showed two distinct choice strategies. Choices after short inter-trial-intervals (ITIs) depended only on the last trial outcome and followed a win-stay-lose-switch pattern. In contrast, choices after long ITIs reflected outcome history over multiple trials, as described by reinforcement learning models. We found that optogenetic stimulation during a trial significantly boosted the rate of learning that occurred due to the outcome of that trial, but these effects were only exhibited on choices after long ITIs. This suggests that serotonin neurons modulate reinforcement learning rates, and that this influence is masked by alternate, unaffected, decision mechanisms. These results provide insight into the role of serotonin in treating psychiatric disorders, particularly its modulation of neural plasticity and learning.

Serotonin (5-HT) plays many important roles in reward, punishment, patience and beyond, and optogenetic stimulation of 5-HT neurons has not crisply parsed them. The authors report a novel analysis of a reward-based decision-making experiment, and show that 5-HT stimulation increases the learning rate, but only on a select subset of choices.

The neural encoding of information prediction errors during non-instrumental information seeking

In a dynamic world, accurate beliefs about the environment are vital for survival, and individuals should therefore regularly seek out new information with which to update their beliefs. This aspect of behaviour is not well captured by standard theories of decision making, and the neural mechanisms of information seeking remain unclear. One recent theory posits that valuation of information results from representation of informative stimuli within canonical neural reward-processing circuits, even if that information lacks instrumental use. We investigated this question by recording EEG from twenty-three human participants performing a non-instrumental information-seeking task. In this task, participants could pay a monetary cost to receive advance information about the likelihood of receiving reward in a lottery at the end of each trial. Behavioural results showed that participants were willing to incur considerable monetary costs to acquire early but non-instrumental information. Analysis of the event-related potential elicited by informative cues revealed that the feedback-related negativity independently encoded both an information prediction error and a reward prediction error. These findings are consistent with the hypothesis that information seeking results from processing of information within neural reward circuits, and suggests that information may represent a distinct dimension of valuation in decision making under uncertainty.

Paradoxical choice in rats: Subjective valuation and mechanism of choice

Decision-makers benefit from information only when they can use it to guide behavior. However, recent experiments found that pigeons and starlings value information that they cannot use. Here we show that this paradox is also present in rats, and explore the underlying decision process. Subjects chose between two options that delivered food probabilistically after a fixed delay. In one option ("info"), outcomes (food/no-food) were signaled immediately after choice, whereas in the alternative ("non-info") the outcome was uncertain until the delay lapsed. Rats sacrificed up to 20% potential rewards by preferring the info option, but reversed preference when the cost was 60%. This reversal contrasts with the results found with pigeons and starlings and may reflect species' differences worth of further investigation. Results are consistent with predictions of the Sequential Choice Model (SCM), that proposes that choices are driven by the mechanisms that control action in sequential encounters. As expected from the SCM, latencies to respond in single-option trials predicted preferences in choice trials, and latencies in choice trials were the same or shorter than in single-option trials. We argue that the congruence of results in distant vertebrates probably reflects evolved adaptations to shared fundamental challenges in nature, and that the apparently paradoxical overvaluing of information is not sub-optimal as has been claimed, even though its functional significance is not yet understood.

Motivated Cognition: Effects of Reward, Emotion, and Other Motivational Factors Across a Variety of Cognitive Domains

A growing body of literature has demonstrated that motivation influences cognitive processing. The breadth of these effects is extensive and span influences of reward, emotion, and other motivational processes across all cognitive domains. As examples, this scope includes studies of emotional memory, value-based attentional capture, emotion effects on semantic processing, reward-related biases in decision making, and the role of approach/avoidance motivation on cognitive scope. Additionally, other less common forms of motivation–cognition interactions, such as self-referential and motoric processing can also be considered instances of motivated cognition. Here I outline some of the evidence indicating the generality and pervasiveness of these motivation influences on cognition, and introduce the associated ‘research nexus’ at Collabra: Psychology.

Understanding active sampling strategies: Empirical approaches and implications for attention and decision research

In natural behavior we actively gather information using attention and active sensing behaviors (such as shifts of gaze) to sample relevant cues. However, while attention and decision making are naturally coordinated, in the laboratory they have been dissociated. Attention is studied independently of the actions it serves. Conversely, decision theories make the simplifying assumption that the relevant information is given, and do not attempt to describe how the decision maker may learn and implement active sampling policies. In this paper I review recent studies that address questions of attentional learning, cue validity and information seeking in humans and non-human primates. These studies suggest that learning a sampling policy involves large scale interactions between networks of attention and valuation, which implement these policies based on reward maximization, uncertainty reduction and the intrinsic utility of cognitive states. I discuss the importance of using such paradigms for formalizing the role of attention, as well as devising more realistic theories of decision making that capture a broader range of empirical observations.