From conflict management to reward-based decision making: Actors and critics in primate medial frontal cortex

The hierarchical basis of neurovisceral integration

The neurovisceral integration (NVI) model was originally proposed to account for observed relationships between peripheral physiology, cognitive performance, and emotional/physical health. This model has also garnered a considerable amount of empirical support, largely from studies examining cardiac vagal control. However, recent advances in functional neuroanatomy, and in computational neuroscience, have yet to be incorporated into the NVI model. Here we present an updated/expanded version of the NVI model that incorporates these advances. Based on a review of studies of structural/functional anatomy, we first describe an eight-level hierarchy of nervous system structures, and the contribution that each level plausibly makes to vagal control. Second, we review recent work on a class of computational models of brain function known as "predictive coding" models. We illustrate how the computational dynamics of these models, when implemented within our proposed vagal control hierarchy, can increase understanding of the relationship between vagal control and both cognitive performance and emotional/physical health. We conclude by discussing novel implications of this updated NVI model for future research.

Dopamine Selectively Modulates the Outcome of Learning Unnatural Action-Valence Associations

Learning the contingencies between stimulus, action, and outcomes is disrupted in disorders associated with altered dopamine (DA) function in the BG, such as Parkinson disease (PD). Although the role of DA in learning to act has been extensively investigated in PD, the role of DA in "learning to withhold" (or inhibit) action to influence outcomes is not as well understood. The current study investigated the role of DA in learning to act or to withhold action to receive rewarding, or avoid punishing outcomes, in patients with PD tested "off" and "on" dopaminergic medication (n = 19) versus healthy controls (n = 30). Participants performed a reward-based learning task that orthogonalized action and outcome valence (action-reward, inaction-reward, action-punishment, inaction-punishment). We tested whether DA would bias learning toward action, toward reward, or to particular action-outcome interactions. All participants demonstrated inherent learning biases preferring action with reward and inaction to avoid punishment, and this was unaffected by medication. Instead, DA produced a complex modulation of learning less natural action-outcome associations. "Off" DA medication, patients demonstrated impairments in learning to withhold action to gain reward, suggesting a difficulty to overcome a bias toward associating inaction with punishment avoidance. On DA medication, these patterns changed, and patients showed a reduced ability to learn to act to avoid punishment, indicating a bias toward action and reward. The current findings suggest that DA in PD has a complex influence on the formation of action-outcome associations, particularly those involving less natural linkages between action and outcome valence.

The Role of Anterior Cingulate Cortex in the Affective Evaluation of Conflict

An influential theory of ACC function argues that this brain region plays a crucial role in the affective evaluation of performance monitoring and control demands. Specifically, control-demanding processes such as response conflict are thought to be registered as aversive signals by ACC, which in turn triggers processing adjustments to support avoidance learning. In support of conflict being treated as an aversive event, recent behavioral studies demonstrated that incongruent (i.e., conflict inducing), relative to congruent, stimuli can speed up subsequent negative, relative to positive, affective picture processing. Here, we used fMRI to investigate directly whether ACC activity in response to negative versus positive pictures is modulated by preceding control demands, consisting of conflict and task-switching conditions. The results show that negative, relative to positive, pictures elicited higher ACC activation after congruent, relative to incongruent, trials, suggesting that ACC's response to negative (positive) pictures was indeed affectively primed by incongruent (congruent) trials. Interestingly, this pattern of results was observed on task repetitions but disappeared on task alternations. This study supports the proposal that conflict induces negative affect and is the first to show that this affective signal is reflected in ACC activation.

Exposure to blue wavelength light modulates anterior cingulate cortex activation in response to 'uncertain' versus 'certain' anticipation of positive stimuli

Blue wavelength light has been used as an effective treatment for some types of mood disorders and circadian rhythm related sleep problems. We hypothesized that acute exposure to blue wavelength light would directly affect the functioning of neurocircuity implicated in emotion regulation (i.e., ventromedial prefrontal cortex, amygdala, insula, and anterior cingulate cortex [ACC]) during 'certain' and 'uncertain' anticipation of negative and positive stimuli. Thirty-five healthy adults were randomized to receive a thirty-minute exposure to either blue (active) or amber (placebo) light, immediately followed by an emotional anticipation task during functional magnetic resonance imaging (fMRI). In contrast to placebo, participants in the blue light group showed significantly reduced activation within the rostral ACC during 'uncertain' anticipation (i.e., uncertainty regarding whether a positive or negative stimulus would be shown) in comparison to 'certain' anticipation of a positive stimulus. These findings may be explicable in terms of interactions between blue light exposure and the influence of specific neuromodulators on ACC-mediated decision-making mechanisms.

'Forget me (not)?' - Remembering Forget-Items Versus Un-Cued Items in Directed Forgetting

Humans need to be able to selectively control their memories. This capability is often investigated in directed forgetting (DF) paradigms. In item-method DF, individual items are presented and each is followed by either a forget- or remember-instruction. On a surprise test of all items, memory is then worse for to-be-forgotten items (TBF) compared to to-be-remembered items (TBR). This is thought to result mainly from selective rehearsal of TBR, although inhibitory mechanisms also appear to be recruited by this paradigm. Here, we investigate whether the mnemonic consequences of a forget instruction differ from the ones of incidental encoding, where items are presented without a specific memory instruction. Four experiments were conducted where un-cued items (UI) were interspersed and recognition performance was compared between TBR, TBF, and UI stimuli. Accuracy was encouraged via a performance-dependent monetary bonus. Experiments varied the number of items and their presentation speed and used either letter-cues or symbolic cues. Across all experiments, including perceptually fully counterbalanced variants, memory accuracy for TBF was reduced compared to TBR, but better than for UI. Moreover, participants made consistently fewer false alarms and used a very conservative response criterion when responding to TBF stimuli. Thus, the F-cue results in active processing and reduces false alarm rate, but this does not impair recognition memory beyond an un-cued baseline condition, where only incidental encoding occurs. Theoretical implications of these findings are discussed.

Generalization in category learning: the roles of representational and decisional uncertainty

Effective generalization in a multiple-category situation involves both assessing potential membership in individual categories and resolving conflict between categories while implementing a decision bound. We separated generalization from decision bound implementation using an information integration task in which category exemplars varied over two incommensurable feature dimensions. Human subjects first learned to categorize stimuli within limited training regions, and then, during fMRI scanning, they also categorized transfer stimuli from new regions of perceptual space. Transfer stimuli differed both in distance from the training region prototype and distance from the decision bound, allowing us to independently assess neural systems sensitive to each. Across all stimulus regions, categorization was associated with activity in the extrastriate visual cortex, basal ganglia, and the bilateral intraparietal sulcus. Categorizing stimuli near the decision bound was associated with recruitment of the frontoinsular cortex and medial frontal cortex, regions often associated with conflict and which commonly coactivate within the salience network. Generalization was measured in terms of greater distance from the decision bound and greater distance from the category prototype (average training region stimulus). Distance from the decision bound was associated with activity in the superior parietal lobe, lingual gyri, and anterior hippocampus, whereas distance from the prototype was associated with left intraparietal sulcus activity. The results are interpreted as supporting the existence of different uncertainty resolution mechanisms for uncertainty about category membership (representational uncertainty) and uncertainty about decision bound (decisional uncertainty).

Signatures of Value Comparison in Ventral Striatum Neurons

The ventral striatum (VS), like its cortical afferents, is closely associated with processing of rewards, but the relative contributions of striatal and cortical reward systems remains unclear. Most theories posit distinct roles for these structures, despite their similarities. We compared responses of VS neurons to those of ventromedial prefrontal cortex (vmPFC) Area 14 neurons, recorded in a risky choice task. Five major response patterns observed in vmPFC were also observed in VS: (1) offer value encoding, (2) value difference encoding, (3) preferential encoding of chosen relative to unchosen value, (4) a correlation between residual variance in responses and choices, and (5) prominent encoding of outcomes. We did observe some differences as well; in particular, preferential encoding of the chosen option was stronger and started earlier in VS than in vmPFC. Nonetheless, the close match between vmPFC and VS suggests that cortex and its striatal targets make overlapping contributions to economic choice.

A study of single neurons in the ventral striatum reveals signatures of value comparison and selection during a risky choice task, suggesting that the cortex and its striatal targets make overlapping contributions to the choice process. Read the accompanying Primer.

The neural calculations underlying reward-based choice are closely associated with a network of brain areas including the ventral striatum (VS) and ventromedial prefrontal cortex (vmPFC). Most theories ascribe distinct roles to these two structures during choice, but these differences have yet to be confirmed at the level of single neurons. We compared responses of VS neurons to those of vmPFC neurons recorded in rhesus macaques choosing between potential gambles for water rewards. We found widespread similarities in the way that VS and vmPFC neurons fire during the choice process. Neurons in both areas encoded the value of the offered gamble, the difference in value between offered gambles, and the gamble outcome. Additionally, both areas showed stronger coding for the chosen gamble than for the unchosen one and predicted choice even when we controlled for offer value. Interestingly, preferential encoding of the chosen option was stronger and started earlier in VS than in vmPFC. Nonetheless, similarities between vmPFC and VS suggest that cortex and its striatal targets make overlapping contributions to reward-based choice.

Vicarious reinforcement learning signals when instructing others

Reinforcement learning (RL) theory posits that learning is driven by discrepancies between the predicted and actual outcomes of actions (prediction errors [PEs]). In social environments, learning is often guided by similar RL mechanisms. For example, teachers monitor the actions of students and provide feedback to them. This feedback evokes PEs in students that guide their learning. We report the first study that investigates the neural mechanisms that underpin RL signals in the brain of a teacher. Neurons in the anterior cingulate cortex (ACC) signal PEs when learning from the outcomes of one's own actions but also signal information when outcomes are received by others. Does a teacher's ACC signal PEs when monitoring a student's learning? Using fMRI, we studied brain activity in human subjects (teachers) as they taught a confederate (student) action-outcome associations by providing positive or negative feedback. We examined activity time-locked to the students' responses, when teachers infer student predictions and know actual outcomes. We fitted a RL-based computational model to the behavior of the student to characterize their learning, and examined whether a teacher's ACC signals when a student's predictions are wrong. In line with our hypothesis, activity in the teacher's ACC covaried with the PE values in the model. Additionally, activity in the teacher's insula and ventromedial prefrontal cortex covaried with the predicted value according to the student. Our findings highlight that the ACC signals PEs vicariously for others' erroneous predictions, when monitoring and instructing their learning. These results suggest that RL mechanisms, processed vicariously, may underpin and facilitate teaching behaviors.

Adaptive effort investment in cognitive and physical tasks: a neurocomputational model

Despite its importance in everyday life, the computational nature of effort investment remains poorly understood. We propose an effort model obtained from optimality considerations, and a neurocomputational approximation to the optimal model. Both are couched in the framework of reinforcement learning. It is shown that choosing when or when not to exert effort can be adaptively learned, depending on rewards, costs, and task difficulty. In the neurocomputational model, the limbic loop comprising anterior cingulate cortex (ACC) and ventral striatum in the basal ganglia allocates effort to cortical stimulus-action pathways whenever this is valuable. We demonstrate that the model approximates optimality. Next, we consider two hallmark effects from the cognitive control literature, namely proportion congruency and sequential congruency effects. It is shown that the model exerts both proactive and reactive cognitive control. Then, we simulate two physical effort tasks. In line with empirical work, impairing the model's dopaminergic pathway leads to apathetic behavior. Thus, we conceptually unify the exertion of cognitive and physical effort, studied across a variety of literatures (e.g., motivation and cognitive control) and animal species.

Feedback-driven trial-by-trial learning in autism spectrum disorders

OBJECTIVE

Impairments in learning are central to autism spectrum disorders. The authors investigated the cognitive and neural basis of these deficits in young adults with autism spectrum disorders using a well-characterized probabilistic reinforcement learning paradigm.

METHOD

The probabilistic selection task was implemented among matched participants with autism spectrum disorders (N=22) and with typical development (N=25), aged 18-40 years, using rapid event-related functional MRI. Participants were trained to choose the correct stimulus in high-probability (AB), medium-probability (CD), and low-probability (EF) pairs, presented with valid feedback 80%, 70%, and 60% of the time, respectively. Whole-brain voxel-wise and parametric modulator analyses examined early and late learning during the stimulus and feedback epochs of the task.

RESULTS

The groups exhibited comparable performance on medium- and low-probability pairs. Typically developing persons showed higher accuracy on the high-probability pair, better win-stay performance (selection of the previously rewarded stimulus on the next trial of that type), and more robust recruitment of the anterior and medial prefrontal cortex during the stimulus epoch, suggesting development of an intact reward-based working memory for recent stimulus values. Throughout the feedback epoch, individuals with autism spectrum disorders exhibited greater recruitment of the anterior cingulate and orbito-frontal cortices compared with individuals with typical development, indicating continuing trial-by-trial activity related to feedback processing.

CONCLUSIONS

Individuals with autism spectrum disorders exhibit learning deficits reflecting impaired ability to develop an effective reward-based working memory to guide stimulus selection. Instead, they continue to rely on trial-by-trial feedback processing to support learning dependent upon engagement of the anterior cingulate and orbito-frontal cortices.

The anterior cingulate gyrus signals the net value of others' rewards

Evaluating the costs and benefits of our own choices is central to most forms of decision-making and its mechanisms in the brain are becoming increasingly well understood. To interact successfully in social environments, it is also essential to monitor the rewards that others receive. Previous studies in nonhuman primates have found neurons in the anterior cingulate cortex (ACC) that signal the net value (benefit minus cost) of rewards that will be received oneself and also neurons that signal when a reward will be received by someone else. However, little is understood about the way in which the human brain engages in cost-benefit analyses during social interactions. Does the ACC signal the net value (the benefits minus the costs) of rewards that others will receive? Here, using fMRI, we examined activity time locked to cues that signaled the anticipated reward magnitude (benefit) to be gained and the level of effort (cost) to be incurred either by a subject themselves or by a social confederate. We investigated whether activity in the ACC covaries with the net value of rewards that someone else will receive when that person is required to exert effort for the reward. We show that, although activation in the sulcus of the ACC signaled the costs on all trials, gyral ACC (ACC(g)) activity varied parametrically only with the net value of rewards gained by others. These results suggest that the ACC(g) plays an important role in signaling cost-benefit information by signaling the value of others' rewards during social interactions.

The role of the midcingulate cortex in monitoring others' decisions

A plethora of research has implicated the cingulate cortex in the processing of social information (i.e., processing elicited by, about, and directed toward others) and reward-related information that guides decision-making. However, it is often overlooked that there is variability in the cytoarchitectonic properties and anatomical connections across the cingulate cortex, which is indicative of functional variability. Here we review evidence from lesion, single-unit recording and functional imaging studies. Taken together, these support the claim that the processing of information that has the greatest influence on social behavior can be localized to the gyral surface of the midcingulate cortex (MCC_g). We propose that the MCC_g is engaged when predicting and monitoring the outcomes of decisions during social interactions. In particular, the MCC_g processes statistical information that tracks the extent to which the outcomes of decisions meet goals when interacting with others. We provide a novel framework for the computational mechanisms that underpin such social information processing in the MCC_g. This framework provides testable hypotheses for the social deficits displayed in autism spectrum disorders and psychopathy.