Stimulus-outcome learnability differentially activates anterior cingulate and hippocampus at feedback processing

Learning under social versus nonsocial uncertainty: A meta-analytic approach

Much of the uncertainty that clouds our understanding of the world springs from the covert values and intentions held by other people. Thus, it is plausible that specialized mechanisms that compute learning signals under uncertainty of exclusively social origin operate in the brain. To test this hypothesis, we scoured academic databases for neuroimaging studies involving learning under uncertainty, and performed a meta‐analysis of brain activation maps that compared learning in the face of social versus nonsocial uncertainty. Although most of the brain activations associated with learning error signals were shared between social and nonsocial conditions, we found some evidence for functional segregation of error signals of exclusively social origin during learning in limited regions of ventrolateral prefrontal cortex and insula. This suggests that most behavioral adaptations to navigate social environments are reused from frontal and subcortical areas processing generic value representation and learning, but that a specialized circuitry might have evolved in prefrontal regions to deal with social context representation and strategic action.

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta-analysis

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Dopaminergic medication impairs feedback-based stimulus-response learning but not response selection in Parkinson's disease

Cognitive dysfunction is a feature of Parkinson's Disease (PD). Some cognitive functions are impaired by dopaminergic medications prescribed to address the movement symptoms that typify PD. Learning appears to be the cognitive function most frequently worsened by dopaminergic therapy. However, this result could reflect either impairments in learning (i.e., acquisition of associations among stimuli, responses, and outcomes) or deficits in performance based on learning (e.g., selecting responses). We sought to clarify the specific effects of dopaminergic medication on (a) stimulus-response association learning from outcome feedback and (b) response selection based on learning, in PD. We tested 28 PD patients on and/or off dopaminergic medication along with 32 healthy, age- and education-matched controls. In Session 1, participants learned to associate abstract images with specific key-press responses through trial and error via outcome feedback. In Session 2, participants provided specific responses to abstract images learned in Session 1, without feedback, precluding new feedback-based learning. By separating Sessions 1 and 2 by 24 h, we could distinguish the effect of dopaminergic medication on (a) feedback-based learning and response selection processes in Session 1 as well as on (b) response selection processes when feedback-based learning could not occur in Session 2. Accuracy achieved at the end of Session 1 were comparable across groups. PD patients on medication learned stimulus-response associations more poorly than PD patients off medication and controls. Medication did not influence decision performance in Session 2. We confirm that dopaminergic therapy impairs feedback-based learning in PD, discounting an alternative explanation that warranted consideration.

Prediction errors to emotional expressions: the roles of the amygdala in social referencing

Social referencing paradigms in humans and observational learning paradigms in animals suggest that emotional expressions are important for communicating valence. It has been proposed that these expressions initiate stimulus-reinforcement learning. Relatively little is known about the role of emotional expressions in reinforcement learning, particularly in the context of social referencing. In this study, we examined object valence learning in the context of a social referencing paradigm. Participants viewed objects and faces that turned toward the objects and displayed a fearful, happy or neutral reaction to them, while judging the gender of these faces. Notably, amygdala activation was larger when the expressions following an object were less expected. Moreover, when asked, participants were both more likely to want to approach, and showed stronger amygdala responses to, objects associated with happy relative to objects associated with fearful expressions. This suggests that the amygdala plays two roles in social referencing: (i) initiating learning regarding the valence of an object as a function of prediction errors to expressions displayed toward this object and (ii) orchestrating an emotional response to the object when value judgments are being made regarding this object.

Co-activation of NR2A and NR2B subunits induces resistance to fear extinction

Unpredictable stress is known to profoundly enhance susceptibility to fear and anxiety while reducing the ability to extinguish fear when threat is no longer present. Accordingly, partial aversive reinforcement, via random exposure to footshocks, induces fear that is resistant to extinction. Here we sought to determine the hippocampal mechanisms underlying susceptibility versus resistance to context fear extinction as a result of continuous (CR) and partial (PR) reinforcement, respectively. We focused on N-methyl-D-aspartate receptor (NMDAR) subunits 2A and B (NR2A and NR2B) as well as their downstream signaling effector, extracellular signal-regulated kinase (ERK), based on their critical role in the acquisition and extinction of fear. Pharmacological inactivation of NR2A, but not NR2B, blocked extinction after CR, whereas inactivation of NR2A, NR2B, or both subunits facilitated extinction after PR. The latter finding suggests that co-activation of NR2A and NR2B contributes to persistent fear following PR. In contrast to CR, PR increased membrane levels of ERK and NR2 subunits after the conditioning and extinction sessions, respectively. In parallel, nuclear activation of ERK was significantly reduced after the extinction session. Thus, co-activation and increased surface expression of NR2A and NR2B, possibly mediated by ERK, may cause persistent fear. These findings suggest that patients with post-traumatic stress disorder (PTSD) may benefit from antagonism of specific NR2 subunits.

Hippocampal activation during extinction learning predicts occurrence of the renewal effect in extinction recall

The renewal effect describes the reoccurrence of a previously extinguished response in situations where the context of extinction differs from that of acquisition, thus illustrating the context-dependency of extinction learning. A number of studies on contextual fear extinction have implicated hippocampus and vmPFC in processing and retrieval of context both during extinction learning and recall of extinction. In this functional magnetic resonance imaging (fMRI) study we explored the neural correlates of the renewal effect in associative learning, using a predictive learning task that required participants to learn relations between cues and outcomes presented in particular contexts. During extinction in a novel context, compared to extinction in a context identical to the acquisition context, participants who exhibited the renewal effect (REN) showed increased activation in brain regions including bilateral posterior hippocampus and left parahippocampal gyrus. This activation pattern was absent in participants that did not show the renewal effect (NOREN). In direct comparisons between the groups, the REN group exhibited higher activation in bilateral hippocampus, while the NOREN group showed higher activation in left dlPFC (BA 46) and right anterior cingulate (BA 32). During extinction recall, stimuli that had been extinguished in a different context were again presented in the context of acquisition. Here both groups exhibited predominantly prefrontal activation, with the REN group's focus upon bilateral OFC (BA 47) and bilateral vmPFC (BA 10), while the NOREN group showed generally more widespread activation, predominantly in large clusters of dlPFC (BA 8,9,45). In a direct comparison, the REN group showed higher activation than the NOREN group in left vmPFC (BA 10), while NOREN participants exhibited more activation in dlPFC (BA 9, 46). Activation in left vmPFC during extinction recall correlated with the number of renewal effect responses, while the dlPFC activation showed a negative correlation with renewal effect responses. These results highlight the differential activation patterns of processes that will eventually produce or not produce a renewal effect, indicating that during extinction learning hippocampus encodes the relation between context and cue-outcome, while in extinction recall vmPFC is active to retrieve this association.

Using conditional maximization to determine hyperparameters in model-based fMRI

In model-based analysis of fMRI data, a neural or cognitive mathematical model of behavior is used to predict changes in fMRI activity. The model predictions are often applied as a parametric modulation of the main stimulus effect within the context of the general linear model (GLM). Using a mathematical model has become an important method for connecting fMRI signals to behavior because the model represents how stimulus processing leads to behavior, and the parametric modulation represents a specific test about the profile of stimulus-related fMRI activity (for review and discussion, see O'Doherty et al., 2007). However, in some cases the parameters of the mathematical model may be under-determined because there is a range of values that equally well account for behavior, or perhaps an exploratory analysis is desired. Thus, in order to fully gauge the applicability of some mathematical model it would be useful to understand how fMRI analysis depends on those parameters. Here, a conditional maximization procedure is developed to search for parameter values in the mathematical model as hyperparameters in the GLM. Simulations and analysis with real fMRI data show that conditional maximization is an effective and simple procedure for estimating hyperparameters. General recommendations and caveats for using hyperparameters in model-based fMRI analysis are also presented.