Differentiable contributions of human amygdalar subregions in the computations underlying reward and avoidance learning

Amygdala-cortical collaboration in reward learning and decision making

Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.

Reinforcement-learning in fronto-striatal circuits

We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.

Distinct Processing of Aversive Experience in Amygdala Subregions

Background

The amygdala is an anatomically complex medial temporal brain structure whose subregions are considered to serve distinct functions. However, their precise role in mediating human aversive experience remains ill understood.

Methods

We used functional magnetic resonance imaging in 39 healthy volunteers with varying levels of trait anxiety to assess distinct contributions of the basolateral amygdala (BLA) and centromedial amygdala to anticipation and experience of aversive events. Additionally, we examined the relationship between any identified functional subspecialization and measures of subjective reported aversion and trait anxiety.

Results

Our results show that the centromedial amygdala is responsive to aversive outcomes but insensitive to predictive aversive cues. In contrast, the BLA encodes an aversive prediction error that quantifies whether cues and outcomes are worse than expected. A neural representation within the BLA for distinct threat levels was mirrored in self-reported subjective anxiety across individuals. Furthermore, high trait-anxious individuals were characterized by indiscriminately heightened BLA activity in response to aversive cues, regardless of actual threat level.

Conclusions

Our results demonstrate that amygdala subregions are distinctly engaged in processing of aversive experience, with elevated and undifferentiated BLA responses to threat emerging as a potential neurobiological mediator of vulnerability to anxiety disorders.

A Computational Model of Attention Control in Multi-Attribute, Context-Dependent Decision Making

Real-life decisions often require a comparison of multi-attribute options with various benefits and costs, and the evaluation of each option depends partly on the others in the choice set (i.e., the choice context). Although reinforcement learning models have successfully described choice behavior, how to account for multi-attribute information when making a context-dependent decision remains unclear. Here we develop a computational model of attention control that includes context effects on multi-attribute decisions, linking a context-dependent choice model with a reinforcement learning model. The overall model suggests that the distinctiveness of attributes guides an individual's preferences among multi-attribute options via an attention-control mechanism that determines whether choices are selectively biased toward the most distinctive attribute (selective attention) or proportionally distributed based on the relative distinctiveness of attributes (divided attention). To test the model, we conducted a behavioral experiment in rhesus monkeys, in which they made simple multi-attribute decisions over three conditions that manipulated the degree of distinctiveness between alternatives: (1) four foods of different size and calorie; (2) four pieces of the same food in different colors; and (3) four identical pieces of food. The model simulation of the choice behavior captured the preference bias (i.e., overall preference structure) and the choice persistence (repeated choices) in the empirical data, providing evidence for the respective influences of attention and memory on preference bias and choice persistence. Our study provides insights into computations underlying multi-attribute decisions, linking attentional control to decision-making processes.

Predicting change: Approximate inference under explicit representation of temporal structure in changing environments

In our daily lives timing of our actions plays an essential role when we navigate the complex everyday environment. It is an open question though how the representations of the temporal structure of the world influence our behavior. Here we propose a probabilistic model with an explicit representation of state durations which may provide novel insights in how the brain predicts upcoming changes. We illustrate several properties of the behavioral model using a standard reversal learning design and compare its task performance to standard reinforcement learning models. Furthermore, using experimental data, we demonstrate how the model can be applied to identify participants' beliefs about the latent temporal task structure. We found that roughly one quarter of participants seem to have learned the latent temporal structure and used it to anticipate changes, whereas the remaining participants' behavior did not show signs of anticipatory responses, suggesting a lack of precise temporal expectations. We expect that the introduced behavioral model will allow, in future studies, for a systematic investigation of how participants learn the underlying temporal structure of task environments and how these representations shape behavior.

Amygdala real-time functional magnetic resonance imaging neurofeedback for major depressive disorder: A review

Advances in imaging technologies have allowed for the analysis of functional magnetic resonance imaging data in real-time (rtfMRI), leading to the development of neurofeedback (nf) training. This rtfMRI-nf training utilizes functional magnetic resonance imaging (fMRI) tomographic localization capacity to allow a person to see and regulate the localized hemodynamic signal from his or her own brain. In this review, we summarize the results of several studies that have developed and applied neurofeedback training to healthy and depressed individuals with the amygdala as the neurofeedback target and the goal to increase the hemodynamic response during positive autobiographical memory recall. We review these studies and highlight some of the challenges and advances in developing an rtfMRI-nf paradigm for broader use in psychiatric populations. The work described focuses on our line of research aiming to develop the rtfMRI-nf into an intervention, and includes a discussion of the selection of a region of interest for feedback, selecting a control condition, behavioral and cognitive effects of training, and predicting which participants are most likely to respond well to training. While the results of these studies are encouraging and suggest the clinical potential of amygdala rtfMRI-nf in alleviating symptoms of major depressive disorder, larger studies are warranted to confirm its efficacy.

Posterior and prefrontal contributions to the development posttraumatic stress disorder symptom severity: an fMRI study of symptom provocation in acute stress disorder

Acute stress disorder (ASD) is predictive of the development of posttraumatic stress disorder (PTSD). In response to symptom provocation, the exposure to trauma-related pictures, ASD patients showed increased activation of the medial posterior areas of precuneus and posterior cingulate cortex as well as of superior prefrontal cortex in a previous study. The current study aimed at investigating which activated areas are predictive of the development of PTSD. Nineteen ASD patients took part in an fMRI study in which they were shown personalized trauma-related and neutral pictures within 4 weeks of the traumatic event. They were assessed for severity of PTSD 4 weeks later. Activation contrasts between trauma-related and neutral pictures were correlated with subsequent PTSD symptom severity. Greater activation in, among others, right medial precuneus, left retrosplenial cortex, precentral and right superior temporal gyrus as well as less activation in lateral, superior prefrontal and left fusiform gyrus was related to subsequently increased PTSD severity. The results are broadly in line with neural areas related to etiological models of PTSD, namely multisensory associative learning recruiting posterior regions on the one hand and failure to reappraise maladaptive cognitions, thought to involve prefrontal areas, on the other.

In vivo quantification of amygdala subnuclei using 4.7 T fast spin echo imaging

The amygdala (AG) is an almond-shaped heterogeneous structure located in the medial temporal lobe. The majority of previous structural Magnetic Resonance Imaging (MRI) volumetric methods for AG measurement have so far only been able to examine this region as a whole. In order to understand the role of the AG in different neuropsychiatric disorders, it is necessary to understand the functional role of its subnuclei. The main goal of the present study was to develop a reliable volumetric method to delineate major AG subnuclei groups using ultra-high resolution high field MRI. 38 healthy volunteers (15 males and 23 females, 21-60 years of age) without any history of medical or neuropsychiatric disorders were recruited for this study. Structural MRI datasets were acquired at 4.7 T Varian Inova MRI system using a fast spin echo (FSE) sequence. The AG was manually segmented into its five major anatomical subdivisions: lateral (La), basal (B), accessory basal (AB) nuclei, and cortical (Co) and centromedial (CeM) groups. Inter-(intra-) rater reliability of our novel volumetric method was assessed using intra-class correlation coefficient (ICC) and Dice's Kappa. Our results suggest that reliable measurements of the AG subnuclei can be obtained by image analysts with experience in AG anatomy. We provided a step-by-step segmentation protocol and reported absolute and relative volumes for the AG subnuclei. Our results showed that the basolateral (BLA) complex occupies seventy-eight percent of the total AG volume, while CeM and Co groups occupy twenty-two percent of the total AG volume. Finally, we observed no hemispheric effects and no gender differences in the total AG volume and the volumes of its subnuclei. Future applications of this method will help to understand the selective vulnerability of the AG subnuclei in neurological and psychiatric disorders.

In vivo delineation of subdivisions of the human amygdaloid complex in a high-resolution group template

The nuclei of the human amygdala remain difficult to distinguish in individual subject structural magnetic resonance images. However, interpretation of the amygdala's role in whole brain networks requires accurate localization of functional activity to a particular nucleus or subgroup of nuclei. To address this, high spatial resolution, three-dimensional templates, using joint high accuracy diffeomorphic registration of T1- and T2-weighted structural images from 168 typical adults between 22 and 35 years old released by the Human Connectome Project were constructed. Several internuclear boundaries are clearly visible in these templates, which would otherwise be impossible to delineate in individual subject data. A probabilistic atlas of major nuclei and nuclear groups was constructed in this template space and mapped back to individual spaces by inversion of the individual diffeomorphisms. Group level analyses revealed a slight (∼2%) bias toward larger total amygdala and nuclear volumes in the right hemisphere. No substantial sex or age differences were found in amygdala volumes normalized to total intracranial volume, or subdivision volumes normalized to amygdala volume. The current delineation provides a finer parcellation of the amygdala with more accurate external boundary definition than current histology-based atlases when used in conjunction with high accuracy registration methods, such as diffeomorphic warping. These templates and delineation are intended to be an open and evolving resource for future functional and structural imaging studies of the human amygdala. Hum Brain Mapp 37:3979-3998, 2016. © 2016 Wiley Periodicals, Inc.

Effects of early life stress on amygdala and striatal development

Species-expected caregiving early in life is critical for the normative development and regulation of emotional behavior, the ability to effectively evaluate affective stimuli in the environment, and the ability to sustain social relationships. Severe psychosocial stressors early in life (early life stress; ELS) in the form of the absence of species expected caregiving (i.e., caregiver deprivation), can drastically impact one's social and emotional success, leading to the onset of internalizing illness later in life. Development of the amygdala and striatum, two key regions supporting affective valuation and learning, is significantly affected by ELS, and their altered developmental trajectories have important implications for cognitive, behavioral and socioemotional development. However, an understanding of the impact of ELS on the development of functional interactions between these regions and subsequent behavioral effects is lacking. In this review, we highlight the roles of the amygdala and striatum in affective valuation and learning in maturity and across development. We discuss their function separately as well as their interaction. We highlight evidence across species characterizing how ELS induced changes in the development of the amygdala and striatum mediate subsequent behavioral changes associated with internalizing illness, positing a particular import of the effect of ELS on their interaction.

Amygdala subnuclei response and connectivity during emotional processing

The involvement of the human amygdala in emotion-related processing has been studied using functional magnetic resonance imaging (fMRI) for many years. However, despite the amygdala being comprised of several subnuclei, most studies investigated the role of the entire amygdala in processing of emotions. Here we combined a novel anatomical tracing protocol with event-related high-resolution fMRI acquisition to study the responsiveness of the amygdala subnuclei to negative emotional stimuli and to examine intra-amygdala functional connectivity. The greatest sensitivity to the negative emotional stimuli was observed in the centromedial amygdala, where the hemodynamic response amplitude elicited by the negative emotional stimuli was greater and peaked later than for neutral stimuli. Connectivity patterns converge with extant findings in animals, such that the centromedial amygdala was more connected with the nuclei of the basal amygdala than with the lateral amygdala. Current findings provide evidence of functional specialization within the human amygdala.

Dissociable Learning Processes Underlie Human Pain Conditioning

Pavlovian conditioning underlies many aspects of pain behavior, including fear and threat detection [1], escape and avoidance learning [2], and endogenous analgesia [3]. Although a central role for the amygdala is well established [4], both human and animal studies implicate other brain regions in learning, notably ventral striatum and cerebellum [5]. It remains unclear whether these regions make different contributions to a single aversive learning process or represent independent learning mechanisms that interact to generate the expression of pain-related behavior. We designed a human parallel aversive conditioning paradigm in which different Pavlovian visual cues probabilistically predicted thermal pain primarily to either the left or right arm and studied the acquisition of conditioned Pavlovian responses using combined physiological recordings and fMRI. Using computational modeling based on reinforcement learning theory, we found that conditioning involves two distinct types of learning process. First, a non-specific “preparatory” system learns aversive facial expressions and autonomic responses such as skin conductance. The associated learning signals—the learned associability and prediction error—were correlated with fMRI brain responses in amygdala-striatal regions, corresponding to the classic aversive (fear) learning circuit. Second, a specific lateralized system learns “consummatory” limb-withdrawal responses, detectable with electromyography of the arm to which pain is predicted. Its related learned associability was correlated with responses in ipsilateral cerebellar cortex, suggesting a novel computational role for the cerebellum in pain. In conclusion, our results show that the overall phenotype of conditioned pain behavior depends on two dissociable reinforcement learning circuits.

•

Different brain learning systems are associated with different defensive responses

•

Cerebellar responses correlate with “associability” for ipsilateral predicted pain

•

The overall phenotype of conditioned pain is the sum of two part-independent processes

Probabilistic reward- and punishment-based learning in opioid addiction: Experimental and computational data

Addiction is the continuation of a habit in spite of negative consequences. A vast literature gives evidence that this poor decision-making behavior in individuals addicted to drugs also generalizes to laboratory decision making tasks, suggesting that the impairment in decision-making is not limited to decisions about taking drugs. In the current experiment, opioid-addicted individuals and matched controls with no history of illicit drug use were administered a probabilistic classification task that embeds both reward-based and punishment-based learning trials, and a computational model of decision making was applied to understand the mechanisms describing individuals' performance on the task. Although behavioral results showed that opioid-addicted individuals performed as well as controls on both reward- and punishment-based learning, the modeling results suggested subtle differences in how decisions were made between the two groups. Specifically, the opioid-addicted group showed decreased tendency to repeat prior responses, meaning that they were more likely to "chase reward" when expectancies were violated, whereas controls were more likely to stick with a previously-successful response rule, despite occasional expectancy violations. This tendency to chase short-term reward, potentially at the expense of developing rules that maximize reward over the long term, may be a contributing factor to opioid addiction. Further work is indicated to better understand whether this tendency arises as a result of brain changes in the wake of continued opioid use/abuse, or might be a pre-existing factor that may contribute to risk for addiction.

The ecology of human fear: survival optimization and the nervous system

We propose a Survival Optimization System (SOS) to account for the strategies that humans and other animals use to defend against recurring and novel threats. The SOS attempts to merge ecological models that define a repertoire of contextually relevant threat induced survival behaviors with contemporary approaches to human affective science. We first propose that the goal of the nervous system is to reduce surprise and optimize actions by (i) predicting the sensory landscape by simulating possible encounters with threat and selecting the appropriate pre-encounter action and (ii) prevention strategies in which the organism manufactures safe environments. When a potential threat is encountered the (iii) threat orienting system is engaged to determine whether the organism ignores the stimulus or switches into a process of (iv) threat assessment, where the organism monitors the stimulus, weighs the threat value, predicts the actions of the threat, searches for safety, and guides behavioral actions crucial to directed escape. When under imminent attack, (v) defensive systems evoke fast reflexive indirect escape behaviors (i.e., fight or flight). This cascade of responses to threat of increasing magnitude are underwritten by an interconnected neural architecture that extends from cortical and hippocampal circuits, to attention, action and threat systems including the amygdala, striatum, and hard-wired defensive systems in the midbrain. The SOS also includes a modulatory feature consisting of cognitive appraisal systems that flexibly guide perception, risk and action. Moreover, personal and vicarious threat encounters fine-tune avoidance behaviors via model-based learning, with higher organisms bridging data to reduce face-to-face encounters with predators. Our model attempts to unify the divergent field of human affective science, proposing a highly integrated nervous system that has evolved to increase the organism's chances of survival.

Processing of action- but not stimulus-related prediction errors differs between active and observational feedback learning

Learning of stimulus-response-outcome associations is driven by outcome prediction errors (PEs). Previous studies have shown larger PE-dependent activity in the striatum for learning from own as compared to observed actions and the following outcomes despite comparable learning rates. We hypothesised that this finding relates primarily to a stronger integration of action and outcome information in active learners. Using functional magnetic resonance imaging, we investigated brain activations related to action-dependent PEs, reflecting the deviation between action values and obtained outcomes, and action-independent PEs, reflecting the deviation between subjective values of response-preceding cues and obtained outcomes. To this end, 16 active and 15 observational learners engaged in a probabilistic learning card-guessing paradigm. On each trial, active learners saw one out of five cues and pressed either a left or right response button to receive feedback (monetary win or loss). Each observational learner observed exactly those cues, responses and outcomes of one active learner. Learning performance was assessed in active test trials without feedback and did not differ between groups. For both types of PEs, activations were found in the globus pallidus, putamen, cerebellum, and insula in active learners. However, only for action-dependent PEs, activations in these structures and the anterior cingulate were increased in active relative to observational learners. Thus, PE-related activity in the reward system is not generally enhanced in active relative to observational learning but only for action-dependent PEs. For the cerebellum, additional activations were found across groups for cue-related uncertainty, thereby emphasising the cerebellum's role in stimulus-outcome learning.

Neural substrates of approach-avoidance conflict decision-making

Animal approach-avoidance conflict paradigms have been used extensively to operationalize anxiety, quantify the effects of anxiolytic agents, and probe the neural basis of fear and anxiety. Results from human neuroimaging studies support that a frontal-striatal-amygdala neural circuitry is important for approach-avoidance learning. However, the neural basis of decision-making is much less clear in this context. Thus, we combined a recently developed human approach-avoidance paradigm with functional magnetic resonance imaging (fMRI) to identify neural substrates underlying approach-avoidance conflict decision-making. Fifteen healthy adults completed the approach-avoidance conflict (AAC) paradigm during fMRI. Analyses of variance were used to compare conflict to nonconflict (avoid-threat and approach-reward) conditions and to compare level of reward points offered during the decision phase. Trial-by-trial amplitude modulation analyses were used to delineate brain areas underlying decision-making in the context of approach/avoidance behavior. Conflict trials as compared to the nonconflict trials elicited greater activation within bilateral anterior cingulate cortex, anterior insula, and caudate, as well as right dorsolateral prefrontal cortex (PFC). Right caudate and lateral PFC activation was modulated by level of reward offered. Individuals who showed greater caudate activation exhibited less approach behavior. On a trial-by-trial basis, greater right lateral PFC activation related to less approach behavior. Taken together, results suggest that the degree of activation within prefrontal-striatal-insula circuitry determines the degree of approach versus avoidance decision-making. Moreover, the degree of caudate and lateral PFC activation related to individual differences in approach-avoidance decision-making. Therefore, the approach-avoidance conflict paradigm is ideally suited to probe anxiety-related processing differences during approach-avoidance decision-making.

Bursts and heavy tails in temporal and sequential dynamics of foraging decisions

A fundamental understanding of behavior requires predicting when and what an individual will choose. However, the actual temporal and sequential dynamics of successive choices made among multiple alternatives remain unclear. In the current study, we tested the hypothesis that there is a general bursting property in both the timing and sequential patterns of foraging decisions. We conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. Regarding when choices were made, we found bursts of rapidly occurring actions, separated by time-varying inactive periods, partially based on a circadian rhythm. Regarding what was chosen, we found sequential dynamics in affective choices characterized by two key features: (a) a highly biased choice distribution; and (b) preferential attachment, in which the animals were more likely to choose what they had previously chosen. To capture the temporal dynamics, we propose a dual-state model consisting of active and inactive states. We also introduce a satiation-attainment process for bursty activity, and a non-homogeneous Poisson process for longer inactivity between bursts. For the sequential dynamics, we propose a dual-control model consisting of goal-directed and habit systems, based on outcome valuation and choice history, respectively. This study provides insights into how the bursty nature of behavior emerges from the interaction of different underlying systems, leading to heavy tails in the distribution of behavior over time and choices.

To understand spontaneous animal behavior, two key elements must be explained: when an action is made and what is chosen. Here, we conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. With respect to when, we found bursts of rapidly occurring responses separated by long inactive periods. With respect to what, we found biased choice behavior toward the favorite items as well as repetitive behavior, reflecting goal-directed and habitual responding, respectively. We account for the when and what components with two distinct computational mechanisms, each composed of two processes: (a) active and inactive states for the temporal dynamics, and (b) goal-directed and habitual control for the sequential dynamics. This study provides behavioral and computational insights into the dynamical properties of decision-making that determine both when an animal will act and what the animal will choose. Our findings provide an integrated framework for describing the temporal and sequential structure of everyday choices among, for example, food, music, books, brands, web-browsing and social interaction.

Free-operant avoidance behavior by rats after reinforcer revaluation using opioid agonists and D-amphetamine

The associative processes that support free-operant instrumental avoidance behavior are still unknown. We used a revaluation procedure to determine whether the performance of an avoidance response is sensitive to the current value of the aversive, negative reinforcer. Rats were trained on an unsignaled, free-operant lever press avoidance paradigm in which each response avoided or escaped shock and produced a 5 s feedback stimulus. The revaluation procedure consisted of noncontingent presentations of the shock in the absence of the lever either paired or unpaired with systemic morphine and in a different cohort with systemic d-amphetamine. Rats were then tested drug free during an extinction test. In both the d-amphetamine and morphine groups, pairing of the drug and shock decreased subsequent avoidance responding during the extinction test, suggesting that avoidance behavior was sensitive to the current incentive value of the aversive negative reinforcer. Experiment 2 used central infusions of D-Ala(2), NMe-Phe(4), Gly-ol(5)]-enkephalin (DAMGO), a mu-opioid receptor agonist, in the periacqueductal gray and nucleus accumbens shell to revalue the shock. Infusions of DAMGO in both regions replicated the effects seen with systemic morphine. These results are the first to demonstrate the impact of revaluation of an aversive reinforcer on avoidance behavior using pharmacological agents, thereby providing potential therapeutic targets for the treatment of avoidance behavior symptomatic of anxiety disorders.

Beep tones attenuate pain following Pavlovian conditioning of an endogenous pain control mechanism

Heterotopic noxious counter-stimulation (HNCS) is commonly used to study endogenous pain control systems. The resulting pain inhibition is primarily based on spinal cord-brainstem loops. Recently, functional imaging studies have shown that limbic structures like the anterior cingulate cortex and amygdala are also implicated. Since these structures are involved in learning processes, it is possible that the HNCS-induced pain inhibition may depend on specific cues from the environment that have been associated with pain reduction through associative learning. We investigated the influence of Pavlovian conditioning on HNCS-induced pain inhibition in 32 healthy subjects by using a differential conditioning paradigm in which two different acoustic stimuli were either repeatedly paired or unpaired with HNCS. Series of noxious electrical pulse trains delivered to the non-dominant foot served as test stimuli. Diffuse noxious inhibitory control (DNIC)-like effects were induced by concurrent application of tonic HNCS (immersion of the contralateral hand in ice water). Subjective pain intensity and pain unpleasantness ratings and electromyographic recordings of the facial corrugator muscle and the nocifensive RIII flexion reflex were used to measure changes in pain sensitivity. HNCS induced significant pain and reflex inhibitions. In the post-conditioning phase, only the paired auditory cue was able to significantly reduce pain perceptions and corrugator muscle activity. No conditioned effect could be observed in RIII reflex responses. Our results indicate that the functional state of endogenous pain control systems may depend on associative learning processes that, like in the present study, may lead to an attenuation of pain perception. Similar albeit opposite conditioning of pain control mechanisms may significantly be involved in the exacerbation and chronification of pain states.

Altered resting-state functional connectivity of basolateral and centromedial amygdala complexes in posttraumatic stress disorder

The amygdala is a major structure that orchestrates defensive reactions to environmental threats and is implicated in hypervigilance and symptoms of heightened arousal in posttraumatic stress disorder (PTSD). The basolateral and centromedial amygdala (CMA) complexes are functionally heterogeneous, with distinct roles in learning and expressing fear behaviors. PTSD differences in amygdala-complex function and functional connectivity with cortical and subcortical structures remain unclear. Recent military veterans with PTSD (n=20) and matched trauma-exposed controls (n=22) underwent a resting-state fMRI scan to measure task-free synchronous blood-oxygen level dependent activity. Whole-brain voxel-wise functional connectivity of basolateral and CMA seeds was compared between groups. The PTSD group had stronger functional connectivity of the basolateral amygdala (BLA) complex with the pregenual anterior cingulate cortex (ACC), dorsomedial prefrontal cortex, and dorsal ACC than the trauma-exposed control group (p<0.05; corrected). The trauma-exposed control group had stronger functional connectivity of the BLA complex with the left inferior frontal gyrus than the PTSD group (p<0.05; corrected). The CMA complex lacked connectivity differences between groups. We found PTSD modulates BLA complex connectivity with prefrontal cortical targets implicated in cognitive control of emotional information, which are central to explanations of core PTSD symptoms. PTSD differences in resting-state connectivity of BLA complex could be biasing processes in target regions that support behaviors central to prevailing laboratory models of PTSD such as associative fear learning. Further research is needed to investigate how differences in functional connectivity of amygdala complexes affect target regions that govern behavior, cognition, and affect in PTSD.

Dissociable patterns of abnormal frontal cortical activation during anticipation of an uncertain reward or loss in bipolar versus major depression

OBJECTIVES

Recent research has found abnormalities in reward-related neural activation in bipolar disorder (BD), during both manic and euthymic phases. However, reward-related neural activation in currently depressed individuals with BD and that in currently depressed individuals with major depressive disorder (MDD) have yet to be directly compared. Here, we studied these groups, examining the neural activation elicited during a guessing task in fronto-striatal regions identified by previous studies.

METHODS

We evaluated neural activation during a reward task using fMRI in two groups of depressed individuals, one with bipolar I disorder (BD-I) (n = 23) and one with MDD (n = 40), with similar levels of illness severity, and a group of healthy individuals (n = 37).

RESULTS

Reward expectancy-related activation in the anterior cingulate cortex was observed in the healthy individuals, but was significantly reduced in depressed patients (BD-I and MDD together). Anticipation-related activation was increased in the left ventrolateral prefrontal cortex in the BD-I depressed group compared with the other two groups. There were no significant differences in prediction error-related activation in the ventral striatum across the three groups.

CONCLUSIONS

The findings extend previous research which has identified dysfunction within the ventrolateral prefrontal cortex in BD, and show that abnormally elevated activity in this region during anticipation of either reward or loss may distinguish depressed individuals with BD-I from those with MDD. Altered activation of the anterior cingulate cortex during reward expectancy characterizes both types of depression. These findings have important implications for identifying both common and distinct properties of the neural circuitry underlying BD-I and MDD.

The neural representation of unexpected uncertainty during value-based decision making

Uncertainty is an inherent property of the environment and a central feature of models of decision-making and learning. Theoretical propositions suggest that one form, unexpected uncertainty, may be used to rapidly adapt to changes in the environment, while being influenced by two other forms: risk and estimation uncertainty. While previous studies have reported neural representations of estimation uncertainty and risk, relatively little is known about unexpected uncertainty. Here, participants performed a decision-making task while undergoing functional magnetic resonance imaging (fMRI), which, in combination with a Bayesian model-based analysis, enabled us to separately examine each form of uncertainty examined. We found representations of unexpected uncertainty in multiple cortical areas, as well as the noradrenergic brainstem nucleus locus coeruleus. Other unique cortical regions were found to encode risk, estimation uncertainty, and learning rate. Collectively, these findings support theoretical models in which several formally separable uncertainty computations determine the speed of learning.

Fiber connectivity between the striatum and cortical and subcortical regions is associated with temperaments in Chinese males

The seven-factor biopsychosocial model of personality distinguished four biologically based temperaments and three psychosocially based characters. Previous studies have suggested that the four temperaments-novelty seeking (NS), reward dependence (RD), harm avoidance (HA), and persistence (P)-have their respective neurobiological correlates, especially in the striatum-connected subcortical and cortical networks. However, few studies have investigated their neurobiological basis in the form of fiber connectivity between brain regions. This study correlated temperaments with fiber connectivity between the striatum and subcortical and cortical hub regions in a sample of 50 Chinese adult males. Generally consistent with our hypotheses, results showed that: (1) NS was positively correlated with fiber connectivity from the medial and lateral orbitofrontal cortex (mOFC, lOFC) and amygdala to the striatum; (2) RD was positively correlated with fiber connectivity from the mOFC, posterior cingulate cortex/retrosplenial cortex (PCC), hippocampus, and amygdala to the striatum; (3) HA was positively linked to fiber connectivity from the dorsolateral prefrontal cortex (dlPFC) and PCC to the striatum; and (4) P was positively linked to fiber connectivity from the mOFC to the striatum. These results extended the research on the neurobiological basis of temperaments by identifying their anatomical fiber connectivity correlates within the subcortical-cortical neural networks.

Pain, decisions, and actions: a motivational perspective

Because pain signals potential harm to the organism, it immediately attracts attention and motivates decisions and action. However, pain is also subject to motivations—an aspect that has led to considerable changes in our understanding of (chronic) pain over the recent years. The relationship between pain and motivational states is therefore clearly bidirectional. This review provides an overview on behavioral and neuroimaging studies investigating motivational aspects of pain. We highlight recent insights into the modulation of pain through fear and social factors, summarize findings on the role of pain in fear conditioning, avoidance learning and goal conflicts and discuss evidence on pain-related cognitive interference and motivational aspects of pain relief.

Distinctive amygdala subregions involved in emotion-modulated Stroop interference

Despite the well-known role of the amygdala in mediating emotional interference during tasks requiring cognitive resources, no definite conclusion has yet been reached regarding the differential roles of functionally and anatomically distinctive subcomponents of the amygdala in such processes. In this study, we examined female participants and attempted to separate the neural processes for the detection of emotional information from those for the regulation of cognitive interference from emotional distractors by adding a temporal gap between emotional stimuli and a subsequent cognitive Stroop task. Reaction time data showed a significantly increased Stroop interference effect following emotionally negative stimuli compared with neutral stimuli, and functional magnetic resonance imaging data revealed that the anterior ventral amygdala (avAMYG) showed greater responses to negative stimuli compared with neutral stimuli. In addition, individuals who scored high in neuroticism showed greater posterior dorsal amygdala (pdAMYG) responses to incongruent compared with congruent Stroop trials following negative stimuli, but not following neutral stimuli. Taken together, the findings of this study demonstrated functionally distinctive contributions of the avAMYG and pdAMYG to the emotion-modulated Stroop interference effect and suggested that the avAMYG encodes associative values of emotional stimuli whereas the pdAMYG resolves cognitive interference from emotional distractors.

To you I am listening: perceived competence of advisors influences judgment and decision-making via recruitment of the amygdala

Considering advice from others is a pervasive element of human social life. We used the judge-advisor paradigm to investigate the neural correlates of advice evaluation and advice integration by means of functional magnetic resonance imaging. Our results demonstrate that evaluating advice recruits the "mentalizing network," brain regions activated when people think about others' mental states. Important activation differences exist, however, depending upon the perceived competence of the advisor. Consistently, additional analyses demonstrate that integrating others' advice, i.e., how much participants actually adjust their initial estimate, correlates with neural activity in the centromedial amygdala in the case of a competent and with activity in visual cortex in the case of an incompetent advisor. Taken together, our findings, therefore, demonstrate that advice evaluation and integration rely on dissociable neural mechanisms and that significant differences exist depending upon the advisor's reputation, which suggests different modes of processing advice depending upon the perceived competence of the advisor.

Evidence for model-based computations in the human amygdala during Pavlovian conditioning

Contemporary computational accounts of instrumental conditioning have emphasized a role for a model-based system in which values are computed with reference to a rich model of the structure of the world, and a model-free system in which values are updated without encoding such structure. Much less studied is the possibility of a similar distinction operating at the level of Pavlovian conditioning. In the present study, we scanned human participants while they participated in a Pavlovian conditioning task with a simple structure while measuring activity in the human amygdala using a high-resolution fMRI protocol. After fitting a model-based algorithm and a variety of model-free algorithms to the fMRI data, we found evidence for the superiority of a model-based algorithm in accounting for activity in the amygdala compared to the model-free counterparts. These findings support an important role for model-based algorithms in describing the processes underpinning Pavlovian conditioning, as well as providing evidence of a role for the human amygdala in model-based inference.

A hot topic in the neurobiology of learning is the idea that there may be two distinct mechanisms for learning in the brain: a model-based learning system in which predictions are made with respect to a rich internal model of the learning environment, versus a “model-free” mechanism in which trial-and-error learning occurs without any rich internal representation of the world. While the focus in the literature to date has been on the role of these mechanisms in instrumental conditioning, almost nothing is known about whether more fundamental kinds of learning such as Pavlovian conditioning also involve model-based processes. Furthermore, nothing is known about the extent to which the amygdala, which is known to be a core structure for Pavlovian learning, contains neural signals consistent with a model-based mechanism. To address this question, we used a novel Pavlovian conditioning task and scanned human volunteers with a special high-resolution fMRI sequence that enabled us to obtain signals within the amygdala with over four times the resolution of conventional imaging protocols. Using this approach in combination with sophisticated computational analyses, we find evidence to suggest that the human amygdala is involved in model-based computations during Pavlovian conditioning.

Separate amygdala subregions signal surprise and predictiveness during associative fear learning in humans

It has recently been suggested that learning signals in the amygdala might be best characterized by attentional theories of associative learning [such as Pearce-Hall (PH)] and more recent hybrid variants that combine Rescorla-Wagner and PH learning models. In these models, unsigned prediction errors (PEs) determine the associability of a cue, which is used in turn to control learning of outcome expectations dynamically and reflects a function of the reliability of prior outcome predictions. Here, we employed an aversive Pavlovian reversal-learning task to investigate computational signals derived from such a hybrid model. Unlike previous accounts, our paradigm allowed for the separate assessment of associability at the time of cue presentation and PEs at the time of outcome. We combined this approach with high-resolution functional magnetic resonance imaging to understand how different subregions of the human amygdala contribute to associative learning. Signal changes in the corticomedial amygdala and in the midbrain represented unsigned PEs at the time of outcome showing increased responses irrespective of whether a shock was unexpectedly administered or omitted. In contrast, activity in basolateral amygdala regions correlated negatively with associability at the time of cue presentation. Thus, whereas the corticomedial amygdala and the midbrain reflected immediate surprise, the basolateral amygdala represented predictiveness and displayed increased responses when outcome predictions became more reliable. These results extend previous findings on PH-like mechanisms in the amygdala and provide unique insights into human amygdala circuits during associative learning.

Neural correlates of specific and general Pavlovian-to-Instrumental Transfer within human amygdalar subregions: a high-resolution fMRI study

It is widely held that the interaction between instrumental and Pavlovian conditioning induces powerful motivational biases. Pavlovian-Instrumental Transfer (PIT) is one of the key paradigms demonstrating this effect, which can further be decomposed into a general and specific component. Although these two forms of PIT have been studied at the level of amygdalar subregions in rodents, it is still unknown whether they involve different areas of the human amygdala. Using a high-resolution fMRI (hr-fMRI) protocol optimized for the amygdala in combination with a novel free operant task designed to elicit effects of both general and specific PIT, we demonstrate that a region of ventral amygdala within the boundaries of the basolateral complex and the ventrolateral putamen are involved in specific PIT, while a region of dorsal amygdala within the boundaries of the centromedial complex is involved in general PIT. These results add to a burgeoning literature indicating different functional contributions for these different amygdalar subregions in reward-processing and motivation.

Threat of punishment motivates memory encoding via amygdala, not midbrain, interactions with the medial temporal lobe

Neural circuits associated with motivated declarative encoding and active threat avoidance have both been described, but the relative contribution of these systems to punishment-motivated encoding remains unknown. The current study used functional magnetic resonance imaging in humans to examine mechanisms of declarative memory enhancement when subjects were motivated to avoid punishments that were contingent on forgetting. A motivational cue on each trial informed participants whether they would be punished or not for forgetting an upcoming scene image. Items associated with the threat of shock were better recognized 24 h later. Punishment-motivated enhancements in subsequent memory were associated with anticipatory activation of right amygdala and increases in its functional connectivity with parahippocampal and orbitofrontal cortices. On a trial-by-trial basis, right amygdala activation during the motivational cue predicted hippocampal activation during encoding of the subsequent scene; across participants, the strength of this interaction predicted memory advantages due to motivation. Of note, punishment-motivated learning was not associated with activation of dopaminergic midbrain, as would be predicted by valence-independent models of motivation to learn. These data are consistent with the view that motivation by punishment activates the amygdala, which in turn prepares the medial temporal lobe for memory formation. The findings further suggest a brain system for declarative learning motivated by punishment that is distinct from that for learning motivated by reward.

Differential neural circuitry and self-interest in real vs hypothetical moral decisions

Classic social psychology studies demonstrate that people can behave in ways that contradict their intentions—especially within the moral domain. We measured brain activity while subjects decided between financial self-benefit (earning money) and preventing physical harm (applying an electric shock) to a confederate under both real and hypothetical conditions. We found a shared neural network associated with empathic concern for both types of decisions. However, hypothetical and real moral decisions also recruited distinct neural circuitry: hypothetical moral decisions mapped closely onto the imagination network, while real moral decisions elicited activity in the bilateral amygdala and anterior cingulate—areas essential for social and affective processes. Moreover, during real moral decision-making, distinct regions of the prefrontal cortex (PFC) determined whether subjects make selfish or pro-social moral choices. Together, these results reveal not only differential neural mechanisms for real and hypothetical moral decisions but also that the nature of real moral decisions can be predicted by dissociable networks within the PFC.

Learning and generalization under ambiguity: an fMRI study

Adaptive behavior often exploits generalizations from past experience by applying them judiciously in new situations. This requires a means of quantifying the relative importance of prior experience and current information, so they can be balanced optimally. In this study, we ask whether the brain generalizes in an optimal way. Specifically, we used Bayesian learning theory and fMRI to test whether neuronal responses reflect context-sensitive changes in ambiguity or uncertainty about experience-dependent beliefs. We found that the hippocampus expresses clear ambiguity-dependent responses that are associated with an augmented rate of learning. These findings suggest candidate neuronal systems that may be involved in aberrations of generalization, such as over-confidence.

Intelligent behavior requires flexible responses to new situations, which exploit learned principles or abstractions. When no such principles exist, the imperative is to learn quickly from scratch. Behaviorally, we show that subjects learn action-reward relationships in a manner that enables them to generalize rules to new situations. Our fMRI results show that when subjects have no evidence that such a rule exists, medial temporal lobe responses (that reflect uncertainty) predict their augmented learning.

Is all motivation good for learning? Dissociable influences of approach and avoidance motivation in declarative memory

The present study investigated the effects of approach versus avoidance motivation on declarative learning. Human participants navigated a virtual reality version of the Morris water task, a classic spatial memory paradigm, adapted to permit the experimental manipulation of motivation during learning. During this task, participants were instructed to navigate to correct platforms while avoiding incorrect platforms. To manipulate motivational states participants were either rewarded for navigating to correct locations (approach) or punished for navigating to incorrect platforms (avoidance). Participants' skin conductance levels (SCLs) were recorded during navigation to investigate the role of physiological arousal in motivated learning. Behavioral results revealed that, overall, approach motivation enhanced and avoidance motivation impaired memory performance compared to nonmotivated spatial learning. This advantage was evident across several performance indices, including accuracy, learning rate, path length, and proximity to platform locations during probe trials. SCL analysis revealed three key findings. First, within subjects, arousal interacted with approach motivation, such that high arousal on a given trial was associated with performance deficits. In addition, across subjects, high arousal negated or reversed the benefits of approach motivation. Finally, low-performing, highly aroused participants showed SCL responses similar to those of avoidance-motivation participants, suggesting that for these individuals, opportunities for reward may evoke states of learning similar to those typically evoked by threats of punishment. These results provide a novel characterization of how approach and avoidance motivation influence declarative memory and indicate a critical and selective role for arousal in determining how reinforcement influences goal-oriented learning.

Identifying a network of brain regions involved in aversion-related processing: a cross-species translational investigation

The ability to detect and respond appropriately to aversive stimuli is essential for all organisms, from fruit flies to humans. This suggests the existence of a core neural network which mediates aversion-related processing. Human imaging studies on aversion have highlighted the involvement of various cortical regions, such as the prefrontal cortex, while animal studies have focused largely on subcortical regions like the periaqueductal gray and hypothalamus. However, whether and how these regions form a core neural network of aversion remains unclear. To help determine this, a translational cross-species investigation in humans (i.e., meta-analysis) and other animals (i.e., systematic review of functional neuroanatomy) was performed. Our results highlighted the recruitment of the anterior cingulate cortex, the anterior insula, and the amygdala as well as other subcortical (e.g., thalamus, midbrain) and cortical (e.g., orbitofrontal) regions in both animals and humans. Importantly, involvement of these regions remained independent of sensory modality. This study provides evidence for a core neural network mediating aversion in both animals and humans. This not only contributes to our understanding of the trans-species neural correlates of aversion but may also carry important implications for psychiatric disorders where abnormal aversive behavior can often be observed.

Human dorsal striatal activity during choice discriminates reinforcement learning behavior from the gambler's fallacy

Reinforcement learning theory has generated substantial interest in neurobiology, particularly because of the resemblance between phasic dopamine and reward prediction errors. Actor-critic theories have been adapted to account for the functions of the striatum, with parts of the dorsal striatum equated to the actor. Here, we specifically test whether the human dorsal striatum--as predicted by an actor-critic instantiation--is used on a trial-to-trial basis at the time of choice to choose in accordance with reinforcement learning theory, as opposed to a competing strategy: the gambler's fallacy. Using a partial-brain functional magnetic resonance imaging scanning protocol focused on the striatum and other ventral brain areas, we found that the dorsal striatum is more active when choosing consistent with reinforcement learning compared with the competing strategy. Moreover, an overlapping area of dorsal striatum along with the ventral striatum was found to be correlated with reward prediction errors at the time of outcome, as predicted by the actor-critic framework. These findings suggest that the same region of dorsal striatum involved in learning stimulus-response associations may contribute to the control of behavior during choice, thereby using those learned associations. Intriguingly, neither reinforcement learning nor the gambler's fallacy conformed to the optimal choice strategy on the specific decision-making task we used. Thus, the dorsal striatum may contribute to the control of behavior according to reinforcement learning even when the prescriptions of such an algorithm are suboptimal in terms of maximizing future rewards.