Context-sensitive valuation and learning

Context effects in cognitive effort evaluation

When given a choice, people will avoid cognitively effortful courses of action because the experience of effort is evaluated as aversive and costly. At the same time, a body of work spanning psychology, economics, and neuroscience suggests that goods, actions, and experiences are often evaluated in the context in which they are encountered, rather in absolute terms. To probe the extent to which the evaluation of cognitive effort is also context-dependent, we had participants learn associations between unique stimuli and subjective demand levels across low-demand and high-demand contexts. We probed demand preferences and subjective evaluation using a forced-choice paradigm as well by examining effort ratings, taken both on-line (during learning) and off-line (after choice). When choosing between two stimuli objectively identical in terms of demand, participants showed a clear preference for the stimulus learned in the low- versus high-demand context and rated this stimulus as more subjectively effortful than the low-demand context in on-line but not off-line ratings, suggesting an assimilation effect. Finally, we observed that the extent to which individual participants who exhibited stronger assimilation effects in off-line demand ratings were more likely to manifest an assimilation effect in demand preferences. Broadly, our findings suggest that effort evaluations occur in a context-dependent manner and are specifically assimilated to the broader context in which they occur.

Comparing experience- and description-based economic preferences across 11 countries

Recent evidence indicates that reward value encoding in humans is highly context dependent, leading to suboptimal decisions in some cases, but whether this computational constraint on valuation is a shared feature of human cognition remains unknown. Here we studied the behaviour of n = 561 individuals from 11 countries of markedly different socioeconomic and cultural makeup. Our findings show that context sensitivity was present in all 11 countries. Suboptimal decisions generated by context manipulation were not explained by risk aversion, as estimated through a separate description-based choice task (that is, lotteries) consisting of matched decision offers. Conversely, risk aversion significantly differed across countries. Overall, our findings suggest that context-dependent reward value encoding is a feature of human cognition that remains consistently present across different countries, as opposed to description-based decision-making, which is more permeable to cultural factors.

The role of the human hippocampus in decision-making under uncertainty

The role of the hippocampus in decision-making is beginning to be more understood. Because of its prospective and inferential functions, we hypothesized that it might be required specifically when decisions involve the evaluation of uncertain values. A group of individuals with autoimmune limbic encephalitis-a condition known to focally affect the hippocampus-were tested on how they evaluate reward against uncertainty compared to reward against another key attribute: physical effort. Across four experiments requiring participants to make trade-offs between reward, uncertainty and effort, patients with acute limbic encephalitis demonstrated blunted sensitivity to reward and effort whenever uncertainty was considered, despite demonstrating intact uncertainty sensitivity. By contrast, the valuation of these two attributes (reward and effort) was intact on uncertainty-free tasks. Reduced sensitivity to changes in reward under uncertainty correlated with the severity of hippocampal damage. Together, these findings provide evidence for a context-sensitive role of the hippocampus in value-based decision-making, apparent specifically under conditions of uncertainty.

Connected in Bad Times and in Good Times: Empathy Induces Stable Social Closeness

Humans need social closeness to prosper. There is evidence that empathy can induce social closeness. However, it remains unclear how empathy-related social closeness is formed and how stable it is as time passes. We applied an acquisition-extinction paradigm combined with computational modeling and fMRI, to investigate the formation and stability of empathy-related social closeness. Female participants observed painful stimulation of another person with high probability (acquisition) and low probability (extinction) and rated their closeness to that person. The results of two independent studies showed increased social closeness in the acquisition block that resisted extinction in the extinction block. Providing insights into underlying mechanisms, reinforcement learning modeling revealed that the formation of social closeness is based on a learning signal (prediction error) generated from observing another's pain, whereas maintaining social closeness is based on a learning signal generated from observing another's pain relief. The results of a reciprocity control study indicate that this feedback recalibration is specific to learning of empathy-related social closeness. On the neural level, the recalibration of the feedback signal was associated with neural responses in anterior insula and adjacent inferior frontal gyrus and the bilateral superior temporal sulcus/temporoparietal junction. Together, these findings show that empathy-related social closeness generated in bad times, that is, empathy with the misfortune of another person, transfers to good times and thus may form one important basis for stable social relationships.

Recent Opioid Use Impedes Range Adaptation in Reinforcement Learning in Human Addiction

BACKGROUND

Drugs like opioids are potent reinforcers thought to co-opt value-based decisions by overshadowing other rewarding outcomes, but how this happens at a neurocomputational level remains elusive. Range adaptation is a canonical process of fine-tuning representations of value based on reward context. Here, we tested whether recent opioid exposure impacts range adaptation in opioid use disorder, potentially explaining why shifting decision making away from drug taking during this vulnerable period is so difficult.

METHODS

Participants who had recently (<90 days) used opioids (n = 34) or who had abstained from opioid use for ≥ 90 days (n = 20) and comparison control participants (n = 44) completed a reinforcement learning task designed to induce robust contextual modulation of value. Two models were used to assess the latent process that participants engaged while making their decisions: 1) a Range model that dynamically tracks context and 2) a standard Absolute model that assumes stationary, objective encoding of value.

RESULTS

Control participants and ≥90-days-abstinent participants with opioid use disorder exhibited choice patterns consistent with range-adapted valuation. In contrast, participants with recent opioid use were more prone to learn and encode value on an absolute scale. Computational modeling confirmed the behavior of most control participants and ≥90-days-abstinent participants with opioid use disorder (75%), but a minority in the recent use group (38%), was better fit by the Range model than the Absolute model. Furthermore, the degree to which participants relied on range adaptation correlated with duration of continuous abstinence and subjective craving/withdrawal.

CONCLUSIONS

Reduced context adaptation to available rewards could explain difficulty deciding about smaller (typically nondrug) rewards in the aftermath of drug exposure.

Social environment-based opportunity costs dictate when people leave social interactions

There is an ever-increasing understanding of the cognitive mechanisms underlying how we process others' behaviours during social interactions. However, little is known about how people decide when to leave an interaction. Are these decisions shaped by alternatives in the environment - the opportunity-costs of connecting to other people? Here, participants chose when to leave partners who treated them with varying degrees of fairness, and connect to others, in social environments with different opportunity-costs. Across four studies we find people leave partners more quickly when opportunity-costs are high, both the average fairness of people in the environment and the effort required to connect to another partner. People's leaving times were accounted for by a fairness-adapted evidence accumulation model, and modulated by depression and loneliness scores. These findings demonstrate the computational processes underlying decisions to leave, and highlight atypical social time allocations as a marker of poor mental health.

Neural and computational underpinnings of biased confidence in human reinforcement learning

While navigating a fundamentally uncertain world, humans and animals constantly evaluate the probability of their decisions, actions or statements being correct. When explicitly elicited, these confidence estimates typically correlates positively with neural activity in a ventromedial-prefrontal (VMPFC) network and negatively in a dorsolateral and dorsomedial prefrontal network. Here, combining fMRI with a reinforcement-learning paradigm, we leverage the fact that humans are more confident in their choices when seeking gains than avoiding losses to reveal a functional dissociation: whereas the dorsal prefrontal network correlates negatively with a condition-specific confidence signal, the VMPFC network positively encodes task-wide confidence signal incorporating the valence-induced bias. Challenging dominant neuro-computational models, we found that decision-related VMPFC activity better correlates with confidence than with option-values inferred from reinforcement-learning models. Altogether, these results identify the VMPFC as a key node in the neuro-computational architecture that builds global feeling-of-confidence signals from latent decision variables and contextual biases during reinforcement-learning.

Modulation of aversive value coding in the vertebrate and invertebrate brain

Avoiding potentially dangerous situations is key for the survival of any organism. Throughout life, animals learn to avoid environments, stimuli or actions that can lead to bodily harm. While the neural bases for appetitive learning, evaluation and value-based decision-making have received much attention, recent studies have revealed more complex computations for aversive signals during learning and decision-making than previously thought. Furthermore, previous experience, internal state and systems level appetitive-aversive interactions seem crucial for learning specific aversive value signals and making appropriate choices. The emergence of novel methodologies (computation analysis coupled with large-scale neuronal recordings, neuronal manipulations at unprecedented resolution offered by genetics, viral strategies and connectomics) has helped to provide novel circuit-based models for aversive (and appetitive) valuation. In this review, we focus on recent vertebrate and invertebrate studies yielding strong evidence that aversive value information can be computed by a multitude of interacting brain regions, and that past experience can modulate future aversive learning and therefore influence value-based decisions.

Anhedonia and sensitivity to punishment in schizophrenia, depression and opiate use disorder

BACKGROUND

From a behavioural perspective anhedonia is defined as diminished interest in the engagement of pleasurable activities. Despite its presence across a range of psychiatric disorders, the cognitive processes that give rise to anhedonia remain unclear.

METHODS

Here we examine whether anhedonia is associated with learning from positive and negative outcomes in patients diagnosed with major depression, schizophrenia and opiate use disorder alongside a healthy control group. Responses in the Wisconsin Card Sorting Test - a task associated with healthy prefrontal cortex function - were fitted to the Attentional Learning Model (ALM) which separates learning from positive and negative feedback.

RESULTS

Learning from punishment, but not from reward, was negatively associated with anhedonia beyond other socio-demographic, cognitive and clinical variables. This impairment in punishment sensitivity was also associated with faster responses following negative feedback, independently of the degree of surprise.

LIMITATIONS

Future studies should test the longitudinal association between punishment sensitivity and anhedonia also in other clinical populations controlling for the effect of specific medications.

CONCLUSIONS

Together the results reveal that anhedonic subjects, because of their negative expectations, are less sensitive to negative feedbacks; this might lead them to persist in actions leading to negative outcomes.

Testing models of context-dependent outcome encoding in reinforcement learning

Previous studies of reinforcement learning (RL) have established that choice outcomes are encoded in a context-dependent fashion. Several computational models have been proposed to explain context-dependent encoding, including reference point centering and range adaptation models. The former assumes that outcomes are centered around a running estimate of the average reward in each choice context, while the latter assumes that outcomes are compared to the minimum reward and then scaled by an estimate of the range of outcomes in each choice context. However, there are other computational mechanisms that can explain context dependence in RL. In the present study, a frequency encoding model is introduced that assumes outcomes are evaluated based on their proportional rank within a sample of recently experienced outcomes from the local context. A range-frequency model is also considered that combines the range adaptation and frequency encoding mechanisms. We conducted two fully incentivized behavioral experiments using choice tasks for which the candidate models make divergent predictions. The results were most consistent with models that incorporate frequency or rank-based encoding. The findings from these experiments deepen our understanding of the underlying computational processes mediating context-dependent outcome encoding in human RL.

Context-independent scaling of neural responses to task difficulty in the multiple-demand network

The multiple-demand (MD) network is sensitive to many aspects of cognitive demand, showing increased activation with more difficult tasks. However, it is currently unknown whether the MD network is modulated by the context in which task difficulty is experienced. Using functional magnetic resonance imaging, we examined MD network responses to low, medium, and high difficulty arithmetic problems within 2 cued contexts, an easy versus a hard set. The results showed that MD activity varied reliably with the absolute difficulty of a problem, independent of the context in which the problem was presented. Similarly, MD activity during task execution was independent of the difficulty of the previous trial. Representational similarity analysis further supported that representational distances in the MD network were consistent with a context-independent code. Finally, we identified several regions outside the MD network that showed context-dependent coding, including the inferior parietal lobule, paracentral lobule, posterior insula, and large areas of the visual cortex. In sum, a cognitive effort is processed by the MD network in a context-independent manner. We suggest that this absolute coding of cognitive demand in the MD network reflects the limited range of task difficulty that can be supported by the cognitive apparatus.

Training diversity promotes absolute-value-guided choice

Many decision-making studies have demonstrated that humans learn either expected values or relative preferences among choice options, yet little is known about what environmental conditions promote one strategy over the other. Here, we test the novel hypothesis that humans adapt the degree to which they form absolute values to the diversity of the learning environment. Since absolute values generalize better to new sets of options, we predicted that the more options a person learns about the more likely they would be to form absolute values. To test this, we designed a multi-day learning experiment comprising twenty learning sessions in which subjects chose among pairs of images each associated with a different probability of reward. We assessed the degree to which subjects formed absolute values and relative preferences by asking them to choose between images they learned about in separate sessions. We found that concurrently learning about more images within a session enhanced absolute-value, and suppressed relative-preference, learning. Conversely, cumulatively pitting each image against a larger number of other images across multiple sessions did not impact the form of learning. These results show that the way humans encode preferences is adapted to the diversity of experiences offered by the immediate learning context.

Bumblebees retrieve only the ordinal ranking of foraging options when comparing memories obtained in distinct settings

Are animals’ preferences determined by absolute memories for options (e.g. reward sizes) or by their remembered ranking (better/worse)? The only studies examining this question suggest humans and starlings utilise memories for both absolute and relative information. We show that bumblebees’ learned preferences are based only on memories of ordinal comparisons. A series of experiments showed that after learning to discriminate pairs of different flowers by sucrose concentration, bumblebees preferred flowers (in novel pairings) with (1) higher ranking over equal absolute reward, (2) higher ranking over higher absolute reward, and (3) identical qualitative ranking but different quantitative ranking equally. Bumblebees used absolute information in order to rank different flowers. However, additional experiments revealed that, even when ranking information was absent (i.e. bees learned one flower at a time), memories for absolute information were lost or could no longer be retrieved after at most 1 hr. Our results illuminate a divergent mechanism for bees (compared to starlings and humans) of learned preferences that may have arisen from different adaptations to their natural environment.

Performance-gated deliberation: A context-adapted strategy in which urgency is opportunity cost

Finding the right amount of deliberation, between insufficient and excessive, is a hard decision making problem that depends on the value we place on our time. Average-reward, putatively encoded by tonic dopamine, serves in existing reinforcement learning theory as the opportunity cost of time, including deliberation time. Importantly, this cost can itself vary with the environmental context and is not trivial to estimate. Here, we propose how the opportunity cost of deliberation can be estimated adaptively on multiple timescales to account for non-stationary contextual factors. We use it in a simple decision-making heuristic based on average-reward reinforcement learning (AR-RL) that we call Performance-Gated Deliberation (PGD). We propose PGD as a strategy used by animals wherein deliberation cost is implemented directly as urgency, a previously characterized neural signal effectively controlling the speed of the decision-making process. We show PGD outperforms AR-RL solutions in explaining behaviour and urgency of non-human primates in a context-varying random walk prediction task and is consistent with relative performance and urgency in a context-varying random dot motion task. We make readily testable predictions for both neural activity and behaviour.

Advances in modeling learning and decision-making in neuroscience

An organism's survival depends on its ability to learn about its environment and to make adaptive decisions in the service of achieving the best possible outcomes in that environment. To study the neural circuits that support these functions, researchers have increasingly relied on models that formalize the computations required to carry them out. Here, we review the recent history of computational modeling of learning and decision-making, and how these models have been used to advance understanding of prefrontal cortex function. We discuss how such models have advanced from their origins in basic algorithms of updating and action selection to increasingly account for complexities in the cognitive processes required for learning and decision-making, and the representations over which they operate. We further discuss how a deeper understanding of the real-world complexities in these computations has shed light on the fundamental constraints on optimal behavior, and on the complex interactions between corticostriatal pathways to determine such behavior. The continuing and rapid development of these models holds great promise for understanding the mechanisms by which animals adapt to their environments, and what leads to maladaptive forms of learning and decision-making within clinical populations.