The computational cost of active information sampling before decision-making under uncertainty

The role of the human hippocampus in decision-making under uncertainty

The role of the hippocampus in decision-making is beginning to be more understood. Because of its prospective and inferential functions, we hypothesized that it might be required specifically when decisions involve the evaluation of uncertain values. A group of individuals with autoimmune limbic encephalitis-a condition known to focally affect the hippocampus-were tested on how they evaluate reward against uncertainty compared to reward against another key attribute: physical effort. Across four experiments requiring participants to make trade-offs between reward, uncertainty and effort, patients with acute limbic encephalitis demonstrated blunted sensitivity to reward and effort whenever uncertainty was considered, despite demonstrating intact uncertainty sensitivity. By contrast, the valuation of these two attributes (reward and effort) was intact on uncertainty-free tasks. Reduced sensitivity to changes in reward under uncertainty correlated with the severity of hippocampal damage. Together, these findings provide evidence for a context-sensitive role of the hippocampus in value-based decision-making, apparent specifically under conditions of uncertainty.

Using Computational Phenotyping to Identify Divergent Strategies for Effort Allocation Across the Psychosis Spectrum

BACKGROUND AND HYPOTHESIS

Disturbances in effort-cost decision-making have been highlighted as a potential transdiagnostic process underpinning negative symptoms in individuals with schizophrenia. However, recent studies using computational phenotyping show that individuals employ a range of strategies to allocate effort, and use of different strategies is associated with unique clinical and cognitive characteristics. Building on prior work in schizophrenia, this study evaluated whether effort allocation strategies differed in individuals with distinct psychotic disorders.

STUDY DESIGN

We applied computational modeling to effort-cost decision-making data obtained from individuals with psychotic disorders (n = 190) who performed the Effort Expenditure for Rewards Task. The sample included 91 individuals with schizophrenia/schizoaffective disorder, 90 individuals with psychotic bipolar disorder, and 52 controls.

STUDY RESULTS

Different effort allocation strategies were observed both across and within different disorders. Relative to individuals with psychotic bipolar disorder, a greater proportion of individuals with schizophrenia/schizoaffective disorder did not use reward value or probability information to guide effort allocation. Furthermore, across disorders, different effort allocation strategies were associated with specific clinical and cognitive features. Those who did not use reward value or probability information to guide effort allocation had more severe positive and negative symptoms, and poorer cognitive and community functioning. In contrast, those who only used reward value information showed a trend toward more severe positive symptoms.

CONCLUSIONS

These findings indicate that similar deficits in effort-cost decision-making may arise from different computational mechanisms across the psychosis spectrum.

Boosting Serotonin Increases Information Gathering by Reducing Subjective Cognitive Costs

Serotonin is implicated in the valuation of aversive costs, such as delay or physical effort. However, its role in governing sensitivity to cognitive effort, for example, deliberation costs during information gathering, is unclear. We show that treatment with a serotonergic antidepressant in healthy human individuals of either sex enhances a willingness to gather information when trying to maximize reward. Using computational modeling, we show this arises from a diminished sensitivity to subjective deliberation costs during the sampling process. This result is consistent with the notion that serotonin alleviates sensitivity to aversive costs in a domain-general fashion, with implications for its potential contribution to a positive impact on motivational deficits in psychiatric disorders.SIGNIFICANCE STATEMENT Gathering information about the world is essential for successfully navigating it. However, sampling information is costly, and we need to balance between gathering too little and too much information. The neurocomputational mechanisms underlying this arbitration between a putative gain, such as reward, and the associated costs, such as allocation of cognitive resources, remain unclear. In this study, we show that week-long daily treatment with a serotonergic antidepressant enhances a willingness to gather information when trying to maximize reward. Computational modeling indicates this arises from a reduced perception of aversive costs, rendering information gathering less cognitively effortful. This finding points to a candidate mechanism by which serotonergic treatment might help alleviate motivational deficits in a range of mental illnesses.

More than a feeling: physiological measures of affect index the integration of effort costs and rewards during anticipatory effort evaluation

The notion that humans avoid effortful action is one of the oldest and most persistent in psychology. Influential theories of effort propose that effort valuations are made according to a cost-benefit trade-off: we tend to invest mental effort only when the benefits outweigh the costs. While these models provide a useful conceptual framework, the affective components of effort valuation remain poorly understood. Here, we examined whether primitive components of affective response-positive and negative valence, captured via facial electromyography (fEMG)-can be used to better understand valuations of cognitive effort. Using an effortful arithmetic task, we find that fEMG activity in the corrugator supercilii-thought to index negative valence-1) tracks the anticipation and exertion of cognitive effort and 2) is attenuated in the presence of high rewards. Together, these results suggest that activity in the corrugator reflects the integration of effort costs and rewards during effortful decision-making.

Good-enough attentional guidance

Theories of attention posit that attentional guidance operates on information held in a target template within memory. The template is often thought to contain veridical target features, akin to a photograph, and to guide attention to objects that match the exact target features. However, recent evidence suggests that attentional guidance is highly flexible and often guided by non-veridical features, a subset of features, or only associated features. We integrate these findings and propose that attentional guidance maximizes search efficiency based on a 'good-enough' principle to rapidly localize candidate target objects. Candidates are then serially interrogated to make target-match decisions using more precise information. We suggest that good-enough guidance optimizes the speed-accuracy-effort trade-offs inherent in each stage of visual search.

Emerging Principles of Attention and Information Demand

I review recent literature on information demand and its implications for attention control. I argue that this literature motivates a view of attention as a mechanism that reduces uncertainty by selectively sampling sensory stimuli on the basis of expected information gain (EIG). I discuss emerging evidence on how individuals estimate the two quantities that determine EIG, prior uncertainty and stimulus diagnosticity (predictive accuracy). I also discuss the neural mechanisms that compute EIG and integrate it with rewards in frontoparietal, executive, and neuromodulatory circuits. I end by considering the implications of this framework for a broader understanding of the factors that assign relevance to sensory stimuli and the role of attention in decision making and other cognitive functions.

Active causal structure learning in continuous time

Research on causal cognition has largely focused on learning and reasoning about contingency data aggregated across discrete observations or experiments. However, this setting represents only the tip of the causal cognition iceberg. A more general problem lurking beneath is that of learning the latent causal structure that connects events and actions as they unfold in continuous time. In this paper, we examine how people actively learn about causal structure in a continuous-time setting, focusing on when and where they intervene and how this shapes their learning. Across two experiments, we find that participants' accuracy depends on both the informativeness and evidential complexity of the data they generate. Moreover, participants' intervention choices strike a balance between maximizing expected information and minimizing inferential complexity. People time and target their interventions to create simple yet informative causal dynamics. We discuss how the continuous-time setting challenges existing computational accounts of active causal learning, and argue that metacognitive awareness of one's inferential limitations plays a critical role for successful learning in the wild.

Environmental uncertainty and the advantage of impulsive choice strategies

Choice impulsivity is characterized by the choice of immediate, smaller reward options over future, larger reward options, and is often thought to be associated with negative life outcomes. However, some environments make future rewards more uncertain, and in these environments impulsive choices can be beneficial. Here we examined the conditions under which impulsive vs. non-impulsive decision strategies would be advantageous. We used Markov Decision Processes (MDPs) to model three common decision-making tasks: Temporal Discounting, Information Sampling, and an Explore-Exploit task. We manipulated environmental variables to create circumstances where future outcomes were relatively uncertain. We then manipulated the discount factor of an MDP agent, which affects the value of immediate versus future rewards, to model impulsive and non-impulsive behavior. This allowed us to examine the performance of impulsive and non-impulsive agents in more or less predictable environments. In Temporal Discounting, we manipulated the transition probability to delayed rewards and found that the agent with the lower discount factor (i.e. the impulsive agent) collected more average reward than the agent with a higher discount factor (the non-impulsive agent) by selecting immediate reward options when the probability of receiving the future reward was low. In the Information Sampling task, we manipulated the amount of information obtained with each sample. When sampling led to small information gains, the impulsive MDP agent collected more average reward than the non-impulsive agent. Third, in the Explore-Exploit task, we manipulated the substitution rate for novel options. When the substitution rate was high, the impulsive agent again performed better than the non-impulsive agent, as it explored the novel options less and instead exploited options with known reward values. The results of these analyses show that impulsivity can be advantageous in environments that are unexpectedly uncertain.

Humans trade off search costs and accuracy in a combined visual search and perceptual task

To interact with one's environment, relevant objects have to be selected as targets for saccadic eye movements. Previous studies have demonstrated that factors such as visual saliency and reward influence saccade target selection, and that humans can dynamically trade off these factors to maximize expected value during visual search. However, expected value in everyday situations not only depends on saliency and reward, but also on the required time to find objects, and the likelihood of a successful object-interaction after search. Here we studied whether search costs and the accuracy to discriminate an object feature can be traded off to maximize expected value. We designed a combined visual search and perceptual discrimination task, where participants chose whether to search for an easy- or difficult-to-discriminate target in search displays populated by distractors that shared features with either the easy or the difficult target. Participants received a monetary reward for correct discriminations and were given limited time to complete as many trials as they could. We found that participants considered their discrimination performance and the search costs when choosing targets and, by this, maximized expected value. However, the accumulated reward was constrained by noise in both the choice of which target to search for, and which elements to fixate during search. We conclude that humans take into account the prospective search time and the likelihood of successful a object-interaction, when deciding what to search for. However, search performance is constrained by noise in decisions about what to search for and how to search for it.

Preferences for seeking effort or reward information bias the willingness to work

Research suggests that the temporal order in which people receive information about costs and benefits whilst making decisions can influence their choices. But, do people have a preference for seeking information about costs or benefits when making effort-based decisions, and does this impact motivation? Here, participants made choices about whether to exert different levels of physical effort to obtain different magnitudes of reward, or rest for low reward. Prior to each effort-based choice, they also had to decide which information they wanted to see first: how much physical effort would be required, or how large the reward would be. We found no overall preference for seeking reward or effort information first, but motivation did change when people saw reward or effort information first. Seeking effort information first, both someone's average tendency to do so and their choice to see effort first on a given trial, was associated with reductions in the willingness to exert higher effort. Moreover, the tendency to prefer effort information first was associated with reduced vigorous exercise and higher levels of fatigue in everyday life. These findings highlight that preferences for seeking effort information may be a bias that reduces people's willingness to exert effort in the lab and in everyday life.

Hyperreactivity to uncertainty is a key feature of subjective cognitive impairment

With an increasingly ageing global population, more people are presenting with concerns about their cognitive function, but not all have an underlying neurodegenerative diagnosis. Subjective cognitive impairment (SCI) is a common condition describing self-reported deficits in cognition without objective evidence of cognitive impairment. Many individuals with SCI suffer from depression and anxiety, which have been hypothesised to account for their cognitive complaints. Despite this association between SCI and affective features, the cognitive and brain mechanisms underlying SCI are poorly understood. Here, we show that people with SCI are hyperreactive to uncertainty and that this might be a key mechanism accounting for their affective burden. Twenty-seven individuals with SCI performed an information sampling task, where they could actively gather information prior to decisions. Across different conditions, SCI participants sampled faster and obtained more information than matched controls to resolve uncertainty. Remarkably, despite their ‘urgent’ sampling behaviour, SCI participants were able to maintain their efficiency. Hyperreactivity to uncertainty indexed by this sampling behaviour correlated with the severity of affective burden including depression and anxiety. Analysis of MRI resting functional connectivity revealed that SCI participants had stronger insular-hippocampal connectivity compared to controls, which also correlated with faster sampling. These results suggest that altered uncertainty processing is a key mechanism underlying the psycho-cognitive manifestations in SCI and implicate a specific brain network target for future treatment.

Cognitive Control as a Multivariate Optimization Problem

A hallmark of adaptation in humans and other animals is our ability to control how we think and behave across different settings. Research has characterized the various forms cognitive control can take-including enhancement of goal-relevant information, suppression of goal-irrelevant information, and overall inhibition of potential responses-and has identified computations and neural circuits that underpin this multitude of control types. Studies have also identified a wide range of situations that elicit adjustments in control allocation (e.g., those eliciting signals indicating an error or increased processing conflict), but the rules governing when a given situation will give rise to a given control adjustment remain poorly understood. Significant progress has recently been made on this front by casting the allocation of control as a decision-making problem. This approach has developed unifying and normative models that prescribe when and how a change in incentives and task demands will result in changes in a given form of control. Despite their successes, these models, and the experiments that have been developed to test them, have yet to face their greatest challenge: deciding how to select among the multiplicity of configurations that control can take at any given time. Here, we will lay out the complexities of the inverse problem inherent to cognitive control allocation, and their close parallels to inverse problems within motor control (e.g., choosing between redundant limb movements). We discuss existing solutions to motor control's inverse problems drawn from optimal control theory, which have proposed that effort costs act to regularize actions and transform motor planning into a well-posed problem. These same principles may help shed light on how our brains optimize over complex control configuration, while providing a new normative perspective on the origins of mental effort.