Cognitive strategies regulate fictive, but not reward prediction error signals in a sequential investment task

The road not taken: Common and distinct neural correlates of regret and relief

Humans anticipate and evaluate both obtained and counterfactual outcomes - outcomes that could have been had an alternate decision been taken - and experience associated emotions of regret and relief. Although many functional magnetic resonance imaging (fMRI) studies have examined the neural correlates of these emotions, there is substantial heterogeneity in their results. We conducted coordinate-based ALE and network-based ANM meta-analysis of fMRI studies of experienced regret and relief to examine commonalities and differences in their neural correlates. Regionally, we observed that the experience of both regret and relief was associated with greater activation in the right ventral striatum (VS), which is implicated in tracking reward prediction error. At the network level, regret and relief shared the reward-sensitive mesocorticolimbic network with preferential activation of the medial orbitofrontal cortex (mOFC) for regret processing and medial cingulate cortex (MCC) for relief processing. Our research identified shared and separable brain systems subserving regret and relief experience, which may inform the treatment of regret-related mood disorders.

Reward and fictive prediction error signals in ventral striatum: asymmetry between factual and counterfactual processing

Reward prediction error, the difference between the expected and obtained reward, is known to act as a reinforcement learning neural signal. In the current study, we propose a model fitting approach that combines behavioral and neural data to fit computational models of reinforcement learning. Briefly, we penalized subject-specific fitted parameters that moved away too far from the group median, except when that deviation led to an improvement in the model's fit to neural responses. By means of a probabilistic monetary learning task and fMRI, we compared our approach with standard model fitting methods. Q-learning outperformed actor-critic at both behavioral and neural level, although the inclusion of neuroimaging data into model fitting improved the fit of actor-critic models. We observed both action-value and state-value prediction error signals in the striatum, while standard model fitting approaches failed to capture state-value signals. Finally, left ventral striatum correlated with reward prediction error while right ventral striatum with fictive prediction error, suggesting a functional hemispheric asymmetry regarding prediction-error driven learning.

An ALE Meta-Analysis on Investment Decision-Making

It is claimed that investment decision-making should rely on rational analyses based on facts and not emotions. However, trying to make money out of market forecasts can trigger all types of emotional responses. As the question on how investors decide remains controversial, we carried out an activation likelihood estimation (ALE) meta-analysis using functional magnetic resonance imaging (fMRI) studies that have reported whole-brain analyses on subjects performing an investment task. We identified the ventral striatum, anterior insula, amygdala and anterior cingulate cortex as being involved in this decision-making process. These regions are limbic-related structures which respond to reward, risk and emotional conflict. Our findings support the notion that investment choices are emotional decisions that take into account market information, individual preferences and beliefs.

The effect of emotion regulation on risk-taking and decision-related activity in prefrontal cortex

Emotion regulation impacts the expected emotional responses to the outcomes of risky decisions via activation of cognitive control strategies. However, whether the regulation of emotional responses to preceding, incidental stimuli also impacts risk-taking in subsequent decisions is still poorly understood. In this study, we investigated the interplay between the regulation of incidentally induced emotional responses and subsequent choice behavior using a risky decision-making task in two independent samples (behavioral and functional magnetic resonance imaging experiment). We found that overall, emotion regulation was followed by less risky decisions, which was further reflected in an increase in activation in brain regions in dorsolateral and ventrolateral prefrontal cortex and cingulate cortex. These findings suggest that altering incidental emotions using reappraisal strategies impacts on subsequent risk-taking in decision-making.

Altered behavioral and neural responsiveness to counterfactual gains in the elderly

Counterfactual information processing refers to the consideration of events that did not occur in comparison to those actually experienced, in order to determine optimal actions, and can be formulated as computational learning signals, referred to as fictive prediction errors. Decision making and the neural circuitry for counterfactual processing are altered in healthy elderly adults. This experiment investigated age differences in neural systems for decision making with knowledge of counterfactual outcomes. Two groups of healthy adult participants, young (N = 30; ages 19-30 years) and elderly (N = 19; ages 65-80 years), were scanned with fMRI during 240 trials of a strategic sequential investment task in which a particular strategy of differentially weighting counterfactual gains and losses during valuation is associated with more optimal performance. Elderly participants earned significantly less than young adults, differently weighted counterfactual consequences and exploited task knowledge, and exhibited altered activity in a fronto-striatal circuit while making choices, compared to young adults. The degree to which task knowledge was exploited was positively correlated with modulation of neural activity by expected value in the vmPFC for young adults, but not in the elderly. These findings demonstrate that elderly participants' poor task performance may be related to different counterfactual processing.

Neurocomputational mechanisms of prosocial learning and links to empathy

Reinforcement learning theory powerfully characterizes how we learn to benefit ourselves. In this theory, prediction errors-the difference between a predicted and actual outcome of a choice-drive learning. However, we do not operate in a social vacuum. To behave prosocially we must learn the consequences of our actions for other people. Empathy, the ability to vicariously experience and understand the affect of others, is hypothesized to be a critical facilitator of prosocial behaviors, but the link between empathy and prosocial behavior is still unclear. During functional magnetic resonance imaging (fMRI) participants chose between different stimuli that were probabilistically associated with rewards for themselves (self), another person (prosocial), or no one (control). Using computational modeling, we show that people can learn to obtain rewards for others but do so more slowly than when learning to obtain rewards for themselves. fMRI revealed that activity in a posterior portion of the subgenual anterior cingulate cortex/basal forebrain (sgACC) drives learning only when we are acting in a prosocial context and signals a prosocial prediction error conforming to classical principles of reinforcement learning theory. However, there is also substantial variability in the neural and behavioral efficiency of prosocial learning, which is predicted by trait empathy. More empathic people learn more quickly when benefitting others, and their sgACC response is the most selective for prosocial learning. We thus reveal a computational mechanism driving prosocial learning in humans. This framework could provide insights into atypical prosocial behavior in those with disorders of social cognition.

Belief about Nicotine Modulates Subjective Craving and Insula Activity in Deprived Smokers

Little is known about the specific neural mechanisms through which cognitive factors influence craving and associated brain responses, despite the initial success of cognitive therapies in treating drug addiction. In this study, we investigated how cognitive factors such as beliefs influence subjective craving and neural activities in nicotine-addicted individuals using model-based functional magnetic resonance imaging (fMRI) and neuropharmacology. Deprived smokers (N = 24) participated in a two-by-two balanced placebo design, which crossed beliefs about nicotine (told "nicotine" vs. told "no nicotine") with the nicotine content in a cigarette (nicotine vs. placebo) which participants smoked immediately before performing a fMRI task involving reward learning. Subjects' reported craving was measured both before smoking and after the fMRI session. We found that first, in the presence of nicotine, smokers demonstrated significantly reduced craving after smoking when told "nicotine in cigarette" but showed no change in craving when told "no nicotine." Second, neural activity in the insular cortex related to craving was only significant when smokers were told "nicotine" but not when told "no nicotine." Both effects were absent in the placebo condition. Third, insula activation related to computational learning signals was modulated by belief about nicotine regardless of nicotine's presence. These results suggest that belief about nicotine has a strong impact on subjective craving and insula responses related to both craving and learning in deprived smokers, providing insights into the complex nature of belief-drug interactions.

Necessary, yet dissociable contributions of the insular and ventromedial prefrontal cortices to norm adaptation: computational and lesion evidence in humans

Social norms and their enforcement are fundamental to human societies. The ability to detect deviations from norms and to adapt to norms in a changing environment is therefore important to individuals' normal social functioning. Previous neuroimaging studies have highlighted the involvement of the insular and ventromedial prefrontal (vmPFC) cortices in representing norms. However, the necessity and dissociability of their involvement remain unclear. Using model-based computational modeling and neuropsychological lesion approaches, we examined the contributions of the insula and vmPFC to norm adaptation in seven human patients with focal insula lesions and six patients with focal vmPFC lesions, in comparison with forty neurologically intact controls and six brain-damaged controls. There were three computational signals of interest as participants played a fairness game (ultimatum game): sensitivity to the fairness of offers, sensitivity to deviations from expected norms, and the speed at which people adapt to norms. Significant group differences were assessed using bootstrapping methods. Patients with insula lesions displayed abnormally low adaptation speed to norms, yet detected norm violations with greater sensitivity than controls. Patients with vmPFC lesions did not have such abnormalities, but displayed reduced sensitivity to fairness and were more likely to accept the most unfair offers. These findings provide compelling computational and lesion evidence supporting the necessary, yet dissociable roles of the insula and vmPFC in norm adaptation in humans: the insula is critical for learning to adapt when reality deviates from norm expectations, and that the vmPFC is important for valuation of fairness during social exchange.

Mindfulness meditation modulates reward prediction errors in a passive conditioning task

Reinforcement learning models have demonstrated that phasic activity of dopamine neurons during reward expectation encodes information about the predictability of reward and cues that predict reward. Self-control strategies such as those practiced in mindfulness-based approaches is claimed to reduce negative and positive reactions to stimuli suggesting the hypothesis that such training may influence basic reward processing. Using a passive conditioning task and fMRI in a group of experienced mindfulness meditators and age-matched controls, we tested the hypothesis that mindfulness meditation influence reward and reward prediction error (PE) signals. We found diminished positive and negative PE-related blood-oxygen level-dependent (BOLD) responses in the putamen in meditators compared with controls. In the meditator group this decrease in striatal BOLD responses to reward PE was paralleled by increased activity in posterior insula, a primary interoceptive region. Critically, responses in the putamen during early trials of the conditioning procedure (run 1) were elevated in both meditators and controls. Overall, these results provide evidence that experienced mindfulness meditators are able to attenuate reward prediction signals to valenced stimuli, which may be related to interoceptive processes encoded in the posterior insula.

Belief about nicotine selectively modulates value and reward prediction error signals in smokers

Little is known about how prior beliefs impact biophysically described processes in the presence of neuroactive drugs, which presents a profound challenge to the understanding of the mechanisms and treatments of addiction. We engineered smokers' prior beliefs about the presence of nicotine in a cigarette smoked before a functional magnetic resonance imaging session where subjects carried out a sequential choice task. Using a model-based approach, we show that smokers' beliefs about nicotine specifically modulated learning signals (value and reward prediction error) defined by a computational model of mesolimbic dopamine systems. Belief of "no nicotine in cigarette" (compared with "nicotine in cigarette") strongly diminished neural responses in the striatum to value and reward prediction errors and reduced the impact of both on smokers' choices. These effects of belief could not be explained by global changes in visual attention and were specific to value and reward prediction errors. Thus, by modulating the expression of computationally explicit signals important for valuation and choice, beliefs can override the physical presence of a potent neuroactive compound like nicotine. These selective effects of belief demonstrate that belief can modulate model-based parameters important for learning. The implications of these findings may be far ranging because belief-dependent effects on learning signals could impact a host of other behaviors in addiction as well as in other mental health problems.