Distinct Contributions of Ventromedial and Dorsolateral Subregions of the Human Substantia Nigra to Appetitive and Aversive Learning

Omissions of threat trigger subjective relief and prediction error-like signaling in the human reward and salience systems

The unexpected absence of danger constitutes a pleasurable event that is critical for the learning of safety. Accumulating evidence points to similarities between the processing of absent threat and the well-established reward prediction error (PE). However, clear-cut evidence for this analogy in humans is scarce. In line with recent animal data, we showed that the unexpected omission of (painful) electrical stimulation triggers activations within key regions of the reward and salience pathways and that these activations correlate with the pleasantness of the reported relief. Furthermore, by parametrically violating participants' probability and intensity related expectations of the upcoming stimulation, we showed for the first time in humans that omission-related activations in the VTA/SN were stronger following omissions of more probable and intense stimulations, like a positive reward PE signal. Together, our findings provide additional support for an overlap in the neural processing of absent danger and rewards in humans.

Neural Representation of Valenced and Generic Probability and Uncertainty

Representing the probability and uncertainty of outcomes facilitates adaptive behavior by allowing organisms to prepare in advance and devote attention to relevant events. Probability and uncertainty are often studied only for valenced (appetitive or aversive) outcomes, raising the question of whether the identified neural machinery also processes the probability and uncertainty of motivationally neutral outcomes. Here, we aimed to dissociate valenced from valence-independent (i.e., generic) probability (p; maximum at p = 1) and uncertainty (maximum at p = 0.5) signals using human neuroimaging. In a Pavlovian task (n = 41; 19 females), different cues predicted appetitive, aversive, or neutral liquids with different probabilities (p = 0, p = 0.5, p = 1). Cue-elicited motor responses accelerated, and pupil sizes increased primarily for cues that predicted valenced liquids with higher probability. For neutral liquids, uncertainty rather than probability tended to accelerate cue-induced responding and decrease pupil size. At the neural level, generic uncertainty signals were limited to the occipital cortex, while generic probability also activated the anterior ventromedial prefrontal cortex. These generic probability and uncertainty signals contrasted with cue-induced responses that only encoded the probability and uncertainty of valenced liquids in medial prefrontal, insular, and occipital cortices. Our findings show a behavioral and neural dissociation of generic and valenced signals. Thus, some parts of the brain keep track of motivational charge while others do not, highlighting the need and usefulness of characterizing the exact nature of learned representations.

Pupillary Responses Reflect Dynamic Changes in Multiple Cognitive Factors During Associative Learning in Primates

Associative learning involves complex interactions of multiple cognitive factors. While adult subjects can articulate these factors verbally, for model animals such as macaques, we rely on behavioral outputs. In our study, we used pupillary responses as an alternative measure to capture these underlying cognitive changes. We recorded the dynamic changes in the pupils of three male macaques when they learned the associations between visual stimuli and reward sizes under the classical Pavlovian experimental paradigm. We found that during the long-term learning process, the gradual changes in the pupillary response reflect the changes in the cognitive state of the animals. The pupillary response can be explained by a linear combination of components corresponding to multiple cognitive factors. These components reflect the impact of visual stimuli on the pupils, the prediction of reward values associated with the visual stimuli, and the macaques' understanding of the current experimental reward rules. The changing patterns of these factors during interday and intraday learning clearly demonstrate the enhancement of current reward-stimulus association and the weakening of previous reward-stimulus association. Our study shows that the dynamic response of pupils can serve as an objective indicator to characterize the psychological changes of animals, understand their learning process, and provide important tools for exploring animal behavior during the learning process.

Encoding of Predictive Associations in Human Prefrontal and Medial Temporal Neurons During Pavlovian Appetitive Conditioning

Pavlovian conditioning is thought to involve the formation of learned associations between stimuli and values, and between stimuli and specific features of outcomes. Here, we leveraged human single neuron recordings in ventromedial prefrontal, dorsomedial frontal, hippocampus, and amygdala while patients of both sexes performed an appetitive Pavlovian conditioning task probing both stimulus-value and stimulus-stimulus associations. Ventromedial prefrontal cortex encoded predictive value along with the amygdala, and also encoded predictions about the identity of stimuli that would subsequently be presented, suggesting a role for neurons in this region in encoding predictive information beyond value. Unsigned error signals were found in dorsomedial frontal areas and hippocampus, potentially supporting learning of non-value related outcome features. Our findings implicate distinct human prefrontal and medial temporal neuronal populations in mediating predictive associations which could partially support model-based mechanisms during Pavlovian conditioning.

The spread of affective and semantic valence representations across states

In many decision problems, outcomes are not reached after a single action but rather after a series of events or states. To optimize decisions over multiple states, representations of how good or bad the outcomes are, that is, the outcomes' valence, should spread across states. One mechanism for valence spreading is a temporal, state-independent process in which a single valence representation is updated when an outcome is experienced and fades away afterwards. Each state's valence is based on its temporal proximity to the experienced outcome. An alternative, state-dependent mechanism relies on the structure of transitions between states, updating a separate valence representation for each state according to its spatial distance from the outcomes. We examined how these mechanistic accounts shape the spread of two formats of valence representation, feelings (affective valence) and knowledge (semantic valence), between states. In two pre-registered experiments (N = 585), we used a novel task in which participants move in a four-state maze, one of which contains an outcome. The participants provide self-reports of affective and semantic valence throughout the maze and after finishing it. Results show that the affective representation of negative valence is more localized in state-space than the semantic representation. We also found evidence for the relative reliance of the affective valence on a temporal, state-independent mechanism and of the semantic valence on a structured, state-dependent mechanism. Our findings provide mechanistic accounts for the differences between affective and semantic valence representations and indicate how such representations may play a role in associative learning and decision-making.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans

We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.

Subacute alpha frequency (10Hz) subthalamic stimulation for emotional processing in Parkinson's disease

BACKGROUND

Psychiatric comorbidities are common in Parkinson's disease (PD) and may change with high-frequency stimulation targeting the subthalamic nucleus. Numerous accounts indicate subthalamic alpha-frequency oscillation is implicated in emotional processing. While intermittent alpha-frequency (10Hz) stimulation induces positive emotional effects, with more ventromedial contacts inducing larger effects, little is known about the subacute effect of ventral 10Hz subthalamic stimulation on emotional processing.

OBJECTIVE/HYPOTHESIS

To evaluate the subacute effect of 10Hz stimulation at bilateral ventral subthalamic nucleus on emotional processing in PD patients using an affective task, compared to that of clinical-frequency stimulation and off-stimulation.

METHODS

Twenty PD patients with bilateral subthalamic deep brain stimulation for more than six months were tested with the affective task under three stimulation conditions (10Hz, 130Hz, and off-stimulation) in a double-blinded randomized design.

RESULTS

While 130Hz stimulation reduced arousal ratings in all patients, 10Hz stimulation increased arousal selectively in patients with higher depression scores. Furthermore, 10Hz stimulation induced a positive shift in valence rating to negative emotional stimuli in patients with lower apathy scores, and 130Hz stimulation led to more positive valence to emotional stimuli in the patients with higher apathy scores. Notably, we found correlational relationships between stimulation site and affective rating: arousal ratings increase with stimulation from anterior to posterior site, and positive valence ratings increase with stimulation from dorsal to ventral site of the ventral subthalamic nucleus.

CONCLUSIONS

Our findings highlight the distinctive role of 10Hz stimulation on subjective emotional experience and unveil the spatial organization of the stimulation effect.

Neural responses to reward valence and magnitude from pre- to early adolescence

BACKGROUND

Neural activation during reward processing is thought to underlie critical behavioral changes that take place during the transition to adolescence (e.g., learning, risk-taking). Though literature on the neural basis of reward processing in adolescence is booming, important gaps remain. First, more information is needed regarding changes in functional neuroanatomy in early adolescence. Another gap is understanding whether sensitivity to different aspects of the incentive (e.g., magnitude and valence) changes during the transition into adolescence. We used fMRI from a large sample of preadolescent children to characterize neural responses to incentive valence vs. magnitude during anticipation and feedback, and their change over a period of two years.

METHODS

Data were taken from the Adolescent Cognitive and Brain DevelopmentSM (ABCD®) study release 3.0. Children completed the Monetary Incentive Delay task at baseline (ages 9-10) and year 2 follow-up (ages 11-12). Based on data from two sites (N = 491), we identified activation-based Regions of Interest (ROIs; e.g., striatum, prefrontal regions, etc.) that were sensitive to trial type (win $5, win $0.20, neutral, lose $0.20, lose $5) during anticipation and feedback phases. Then, in an independent subsample (N = 1470), we examined whether these ROIs were sensitive to valence and magnitude and whether that sensitivity changed over two years.

RESULTS

Our results show that most ROIs involved in reward processing (including the striatum, prefrontal cortex, and insula) are specialized, i.e., mainly sensitive to either incentive valence or magnitude, and this sensitivity was consistent over a 2-year period. The effect sizes of time and its interactions were significantly smaller (0.002≤η2≤0.02) than the effect size of trial type (0.06≤η2≤0.30). Interestingly, specialization was moderated by reward processing phase but was stable across development. Biological sex and pubertal status differences were few and inconsistent. Developmental changes were mostly evident during success feedback, where neural reactivity increased over time.

CONCLUSIONS

Our results suggest sub-specialization to valence vs. magnitude within many ROIs of the reward circuitry. Additionally, in line with theoretical models of adolescent development, our results suggest that the ability to benefit from success increases from pre- to early adolescence. These findings can inform educators and clinicians and facilitate empirical research of typical and atypical motivational behaviors during a critical time of development.

Anxiety and depression related abnormalities in socio-affective learning

Affective distress (as observed in anxiety and depression) has been observed to be related to insufficient sensitivity to changing reinforcement during operant learning. Whether such findings are specific to anxiety or depression is unclear given a wider literature relating negative affect to abnormal learning and the possibility that relationships are not consistent across incentive types (i.e., punishment and reward) and outcomes (i.e., positive or negative). In two separate samples (n = 100; n = 88), participants completed an operant learning task with positive or negative, and neutral socio-affective feedback, designed to assess adaptive responses to changing environmental volatility. Individual parameter estimates were generated with hierarchical Bayesian modelling. Effects of manipulations were modelled by decomposing parameters into a linear combination of effects on the logit scale. While effects tended to support prior work, neither general affective distress nor anxiety or depression were consistently related to a decrease in the adaptive adjustment of learning-rates in response to changing environmental volatility (Sample 1: β_{α:volatility} = -0.01, 95 % HDI = -0.14, 0.13; Sample 2: β_{α:volatility} = -0.15, 95 % HDI = -0.37, 0.05). Interaction effects in Sample 1 suggested that while distress was associated with decrements in adaptive learning under punishment-maximisation, it was associated with improvements under reward-maximisation. While our results are broadly consistent with prior work, they suggest that the role of anxiety or depression in volatility learning, if present, is subtle and difficult to detect. Inconsistencies between our samples, along with issues of parameter identifiability complicated interpretation.

Anticipating social feedback involves basal forebrain and mesolimbic functional connectivity

The mesolimbic system and basal forebrain (BF) are implicated in processing rewards and punishment, but their interplay and functional properties of subregions with respect to future social outcomes remain unclear. Therefore, this study investigated regional responses and interregional functional connectivity of the lateral (l), medial (m), and ventral (v) Substantia Nigra (SN), Nucleus Accumbens (NAcc), Nucleus basalis of Meynert (NBM), and Medial Septum/Diagonal Band (MS/DB) during reward and punishment anticipation in a social incentive delay task with neutral, positive, and negative feedback using high-resolution fMRI (1.5mm³). Neuroimaging data (n=36 healthy humans) of the anticipation phase was analyzed using mass-univariate, functional connectivity, and multivariate-pattern analysis. As expected, participants responded faster when anticipating positive and negative compared to neutral social feedback. At the neural level, anticipating social information engaged valence-related and valence-unrelated functional connectivity patterns involving the BF and mesolimbic areas. Precisely, valence-related connectivity between the lSN and NBM was associated with anticipating neutral social feedback, while connectivity between the vSN and NBM was associated with anticipating positive social feedback. A more complex pattern was observed for anticipating negative social feedback, including connectivity between the lSN and MS/DB, lSN and NAcc, as well as mSN and NAcc. To conclude, behavioral responses are modulated by the possibility to obtain positive and avoid negative social feedback. The neural processing of feedback anticipation relies on functional connectivity patterns between the BF and mesolimbic areas associated with the emotional valence of the social information. As such, our findings give novel insights into the underlying neural processes of social information processing.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans

Pavlovian learning depends on multiple and parallel associations leading to distinct classes of conditioned responses that vary in their flexibility following changes in the value of an associated outcome. Here, we aimed to differentiate brain areas involved in learning and encoding associations that are sensitive to changes in the value of an outcome from those that are not sensitive to such changes. To address this question, we combined a Pavlovian learning task with outcome devaluation, eye-tracking and functional magnetic resonance imaging. We used computational modeling to identify brain regions involved in learning stimulus-reward associations and stimulus-stimulus associations, by testing for brain areas correlating with reward-prediction errors and state-prediction errors, respectively. We found that, contrary to theoretical predictions about reward prediction errors being exclusively model-free, voxels correlating with reward prediction errors in the ventral striatum and subgenual anterior cingulate cortex were sensitive to devaluation. On the other hand, brain areas correlating with state prediction errors were found to be devaluation insensitive. In a supplementary analysis, we distinguished brain regions encoding predictions about outcome taste identity from those involved in encoding predictions about its expected spatial location. A subset of regions involved in taste identity predictions were devaluation sensitive while those involved in encoding predictions about spatial location were devaluation insensitive. These findings provide insights into the role of multiple associative mechanisms in the brain in mediating Pavlovian conditioned behavior - illustrating how distinct neural pathways can in parallel produce both devaluation sensitive and devaluation insensitive behaviors.

The association of spouse interactions and emotional learning in interference related to chronic back pain

Social interactions affect individual behaviours, preferences, and attitudes. This is also critical in the context of experiencing pain and expressing pain behaviours, and may relate to learned emotional responses. In this respect, individual variability in the medial prefrontal cortex (mPFC), which is involved in adjusting an organism's behaviour to its environment by evaluating and interpreting information within the context of past experiences, is important. It is critical for selecting suitable behavioural responses within a social environment and may reinforce maladaptation in chronic pain. In our study, we used brain imaging during appetitive and aversive pavlovian conditioning in persons with chronic back pain (CBP), subacute back pain (SABP), and healthy controls (HC), together with information on spouse responses to pain behaviours. We also examined the relationship of these responses with pain-related interference in the patients. Our findings yielded a significant negative association between mPFC responses to appetitive and aversive learning in CBP. We also observed a significant negative association for mPFC responses during aversive learning and distracting spouse responses, and a significant positive association between mPFC responses during appetitive learning and solicitous spouse responses in CBP. Both significantly predicted pain-related interference in the CBP group (explained variance up to 53%). Significant associations were not found for SABP or HC. Our findings support an association between appetitive and aversive pavlovian learning, related brain circuits and spouse responses to pain in CBP, where appetitive and aversive learning processes seem to be differentially involved. This can inform prevention and early intervention in a mechanistic approach.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Asymmetric representation of aversive prediction errors in Pavlovian threat conditioning

Survival in biological environments requires learning associations between predictive sensory cues and threatening outcomes. Such aversive learning may be implemented through reinforcement learning algorithms that are driven by the signed difference between expected and encountered outcomes, termed prediction errors (PEs). While PE-based learning is well established for reward learning, the role of putative PE signals in aversive learning is less clear. Here, we used functional magnetic resonance imaging in humans (21 healthy men and women) to investigate the neural representation of PEs during maintenance of learned aversive associations. Four visual cues, each with a different probability (0, 33, 66, 100%) of being followed by an aversive outcome (electric shock), were repeatedly presented to participants. We found that neural activity at omission (US-) but not occurrence of the aversive outcome (US+) encoded PEs in the medial prefrontal cortex. More expected omission of aversive outcome was associated with lower neural activity. No neural signals fulfilled axiomatic criteria, which specify necessary and sufficient components of PE signals, for signed PE representation in a whole-brain search or in a-priori regions of interest. Our results might suggest that, different from reward learning, aversive learning does not involve signed PE signals that are represented within the same brain region for all conditions.

Cortico-Striatal Activity Characterizes Human Safety Learning via Pavlovian Conditioned Inhibition

Safety learning generates associative links between neutral stimuli and the absence of threat, promoting the inhibition of fear and security-seeking behaviors. Precisely how safety learning is mediated at the level of underlying brain systems, particularly in humans, remains unclear. Here, we integrated a novel Pavlovian conditioned inhibition task with ultra-high field (7 Tesla) fMRI to examine the neural basis of safety learning in 49 healthy participants. In our task, participants were conditioned to two safety signals: a conditioned inhibitor that predicted threat omission when paired with a known threat signal (A+/AX-), and a standard safety signal that generally predicted threat omission (BC-). Both safety signals evoked equivalent autonomic and subjective learning responses but diverged strongly in terms of underlying brain activation (PFDR whole-brain corrected). The conditioned inhibitor was characterized by more prominent activation of the dorsal striatum, anterior insular, and dorsolateral PFC compared with the standard safety signal, whereas the latter evoked greater activation of the ventromedial PFC, posterior cingulate, and hippocampus, among other regions. Further analyses of the conditioned inhibitor indicated that its initial learning was characterized by consistent engagement of dorsal striatal, midbrain, thalamic, premotor, and prefrontal subregions. These findings suggest that safety learning via conditioned inhibition involves a distributed cortico-striatal circuitry, separable from broader cortical regions involved with processing standard safety signals (e.g., CS-). This cortico-striatal system could represent a novel neural substrate of safety learning, underlying the initial generation of "stimulus-safety" associations, distinct from wider cortical correlates of safety processing, which facilitate the behavioral outcomes of learning.SIGNIFICANCE STATEMENT Identifying safety is critical for maintaining adaptive levels of anxiety, but the neural mechanisms of human safety learning remain unclear. Using 7 Tesla fMRI, we compared learning-related brain activity for a conditioned inhibitor, which actively predicted threat omission, and a standard safety signal (CS-), which was passively unpaired with threat. The inhibitor engaged an extended circuitry primarily featuring the dorsal striatum, along with thalamic, midbrain, and premotor/PFC regions. The CS- exclusively involved cortical safety-related regions observed in basic safety conditioning, such as the vmPFC. These findings extend current models to include learning-specific mechanisms for encoding stimulus-safety associations, which might be distinguished from expression-related cortical mechanisms. These insights may suggest novel avenues for targeting dysfunctional safety learning in psychopathology.

Brainstem Mechanisms of Pain Modulation: A within-Subjects 7T fMRI Study of Placebo Analgesic and Nocebo Hyperalgesic Responses

Pain perception can be powerfully influenced by an individual's expectations and beliefs. Although the cortical circuitry responsible for pain modulation has been thoroughly investigated, the brainstem pathways involved in the modulatory phenomena of placebo analgesia and nocebo hyperalgesia remain to be directly addressed. This study used ultra-high-field 7 tesla functional MRI (fMRI) to accurately resolve differences in brainstem circuitry present during the generation of placebo analgesia and nocebo hyperalgesia in healthy human participants (N = 25, 12 male). Over 2 successive days, through blinded application of altered thermal stimuli, participants were deceptively conditioned to believe that two inert creams labeled lidocaine (placebo) and capsaicin (nocebo) were acting to modulate their pain relative to a third Vaseline (control) cream. In a subsequent test phase, fMRI image sets were collected while participants were given identical noxious stimuli to all three cream sites. Pain intensity ratings were collected and placebo and nocebo responses determined. Brainstem-specific fMRI analysis revealed altered activity in key pain modulatory nuclei, including a disparate recruitment of the periaqueductal gray (PAG)-rostral ventromedial medulla (RVM) pathway when both greater placebo and nocebo effects were observed. Additionally, we found that placebo and nocebo responses differentially activated the parabrachial nucleus but overlapped in engagement of the substantia nigra and locus coeruleus. These data reveal that placebo and nocebo effects are generated through differential engagement of the PAG-RVM pathway, which in concert with other brainstem sites likely influences the experience of pain by modulating activity at the level of the dorsal horn.SIGNIFICANCE STATEMENT Understanding endogenous pain modulatory mechanisms would support development of effective clinical treatment strategies for both acute and chronic pain. Specific brainstem nuclei have long been known to play a central role in nociceptive modulation; however, because of the small size and complex organization of the nuclei, previous neuroimaging efforts have been limited in directly identifying how these subcortical networks interact during the development of antinociceptive and pro-nociceptive effects. We used ultra-high-field fMRI to resolve brainstem structures and measure signal change during placebo analgesia and nocebo hyperalgesia. We define overlapping and disparate brainstem circuitry responsible for altering pain perception. These findings extend our understanding of the detailed organization and function of discrete brainstem nuclei involved in pain processing and modulation.

Using Primary Reinforcement to Enhance Translatability of a Human Affect and Decision-Making Judgment Bias Task

Good translatability of behavioral measures of affect (emotion) between human and nonhuman animals is core to comparative studies. The judgment bias (JB) task, which measures "optimistic" and "pessimistic" decision-making under ambiguity as indicators of positive and negative affective valence, has been used in both human and nonhuman animals. However, one key disparity between human and nonhuman studies is that the former typically use secondary reinforcers (e.g., money) whereas the latter typically use primary reinforcers (e.g., food). To address this deficiency and shed further light on JB as a measure of affect, we developed a novel version of a JB task for humans using primary reinforcers. Data on decision-making and reported affective state during the JB task were analyzed using computational modeling. Overall, participants grasped the task well, and as anticipated, their reported affective valence correlated with trial-by-trial variation in offered volume of juice. In addition, previous findings from monetary versions of the task were replicated: More positive prediction errors were associated with more positive affective valence, a higher lapse rate was associated with lower affective arousal, and affective arousal decreased as a function of number of trials completed. There was no evidence that more positive valence was associated with greater "optimism," but instead, there was evidence that affective valence influenced the participants' decision stochasticity, whereas affective arousal tended to influence their propensity for errors. This novel version of the JB task provides a useful tool for investigation of the links between primary reward and punisher experience, affect, and decision-making, especially from a comparative perspective.

Anticipating social incentives recruits alpha-beta oscillations in the human substantia nigra and invigorates behavior across the life span

Anticipating social and non-social incentives recruits shared brain structures and promotes behavior. However, little is known about possible age-related behavioral changes, and how the human substantia nigra (SN) signals positive and negative social information. Therefore, we recorded intracranial electroencephalography (iEEG) from the SN of Parkinson's Disease (PD) patients (n = 12, intraoperative, OFF medication) in combination with a social incentive delay task including photos of neutral, positive or negative human gestures and mimics as feedback. We also tested a group of non-operated PD patients (n = 24, ON and OFF medication), and a sample of healthy young (n = 51) and older (n = 52) adults with behavioral readouts only. Behaviorally, the anticipation of both positive and negative social feedback equally accelerated response times in contrast to neutral social feedback in healthy young and older adults. Although this effect was not significant in the group of operated PD patients - most likely due to the small sample size - iEEG recordings in their SN showed a significant increase in alpha-beta power (9-20 Hz) from 300 to 600 ms after cue onset again for both positive and negative cues. Finally, in non-operated PD patients, the behavioral effect was not modulated by medication status (ON vs OFF medication) suggesting that other processes than dopaminergic neuromodulation play a role in driving invigoration by social incentives. Together, our findings provide novel and direct evidence for a role of the SN in processing positive and negative social information via specific oscillatory mechanisms in the alpha-beta range, and they suggest that anticipating social value in simple cue-outcome associations is intact in healthy aging and PD.

Pupil dilation as an index of Pavlovian conditioning. A systematic review and meta-analysis

The use of pupillometry to track emotional learning processes in humans is generating an increasing interest. Here, we provide a first systematic review and meta-analysis on the value of pupil dilation as a marker of Pavlovian conditioning, focusing on the roles of UCS valence (aversive vs. appetitive), the time course across trials and response intervals within trials. Based on data from 39 independent samples (total n = 1303), our results revealed strong evidence for the overall validity of conditioned pupil responses, with a trend for larger effects in aversive (average g = 0.73) vs. appetitive conditioning (g = 0.39). Response differentiation increased over the course of acquisition. Substantial differentiation effects were found in both early and late response windows. Moderator analyses revealed a consistent influence of UCS modality on differential conditioning, while evidence for moderation by contingency instructions and length of acquisition phase was mixed. The results highlight pupil dilation as a sensitive and reliable index of Pavlovian conditioning across valence categories and stimulus modalities. Important implications regarding methodological considerations and research goals are discussed.

Naloxone precipitated withdrawal increases dopamine release in the dorsal striatum of opioid dependent men

Dopamine (DA) neurotransmission is critical in the neurobiology of reward and aversion, but its contribution to the aversive state of opioid withdrawal remains unknown in humans. To address this, we used updated voxelwise methods and retrospectively analyzed a [¹¹C]raclopride-PET dataset to measure D_2/3 receptor availability and relative cerebral blood flow (R1) in male opioid use disorder (OUD) participants (n = 10) during placebo and acute opioid withdrawal conditions. We found that acute withdrawal precipitated by the opioid antagonist naloxone significantly increased dorsal striatal DA release in OUD participants (p_FWE < 0.05). Net changes in striatal DA were significantly correlated with a subjective index of withdrawal aversion such that greater DA increases were associated with more aversive responses (r(8) = 0.82, p < 0.005). Withdrawal also affected brain function, as indexed by increases in relative cerebral blood flow in the insula and putamen (p_FWE < 0.05). Our findings are different from preclinical studies that have primarily reported decreases in ventral striatal DA during naloxone precipitated withdrawal, whereas this effect was not significant in OUD participants (p = 0.79). In sum, we provide evidence for the contribution of increases in dorsal striatal DA to the aversive state of naloxone precipitated withdrawal in humans.

The Neural Correlates of Cued Reward Omission

Compared to our understanding of positive prediction error signals occurring due to unexpected reward outcomes, less is known about the neural circuitry in humans that drives negative prediction errors during omission of expected rewards. While classical learning theories such as Rescorla-Wagner or temporal difference learning suggest that both types of prediction errors result from a simple subtraction, there has been recent evidence suggesting that different brain regions provide input to dopamine neurons which contributes to specific components of this prediction error computation. Here, we focus on the brain regions responding to negative prediction error signals, which has been well-established in animal studies to involve a distinct pathway through the lateral habenula. We examine the activity of this pathway in humans, using a conditioned inhibition paradigm with high-resolution functional MRI. First, participants learned to associate a sensory stimulus with reward delivery. Then, reward delivery was omitted whenever this stimulus was presented simultaneously with a different sensory stimulus, the conditioned inhibitor (CI). Both reward presentation and the reward-predictive cue activated midbrain dopamine regions, insula and orbitofrontal cortex. While we found significant activity at an uncorrected threshold for the CI in the habenula, consistent with our predictions, it did not survive correction for multiple comparisons and awaits further replication. Additionally, the pallidum and putamen regions of the basal ganglia showed modulations of activity for the inhibitor that did not survive the corrected threshold.

Resting-State Functional Connectivity of the Punishment Network Associated With Conformity

Fear of punishment prompts individuals to conform. However, why some people are more inclined than others to conform despite being unaware of any obvious punishment remains unclear, which means the dispositional determinants of individual differences in conformity propensity are poorly understood. Here, we explored whether such individual differences might be explained by individuals' stable neural markers to potential punishment. To do this, we first defined the punishment network (PN) by combining all potential brain regions involved in punishment processing. We subsequently used a voxel-based global brain connectivity (GBC) method based on resting-state functional connectivity (FC) to characterize the hubs in the PN, which reflected an ongoing readiness state (i.e., sensitivity) for potential punishment. Then, we used the within-network connectivity (WNC) of each voxel in the PN of 264 participants to explain their tendency to conform by using a conformity scale. We found that a stronger WNC in the right thalamus, left insula, postcentral gyrus, and dACC was associated with a stronger tendency to conform. Furthermore, the FC among the four hubs seemed to form a three-phase ascending pathway, contributing to conformity propensity at every phase. Thus, our results suggest that task-independent spontaneous connectivity in the PN could predispose individuals to conform.

Spatiotemporal changes in substantia nigra neuromelanin content in Parkinson’s disease

This study aimed to investigate the spatiotemporal changes in neuromelanin-sensitive MRI signal in the substantia nigra and their relation to clinical scores of disease severity in patients with early or progressing Parkinson’s disease and patients with idiopathic rapid eye movement sleep behaviour disorder (iRBD) exempt of Parkinsonian signs compared to healthy control subjects. Longitudinal T1-weighted anatomical and neuromelanin-sensitive MRI was performed in two cohorts, including patients with iRBD, patients with early or progressing Parkinson’s disease, and control subjects. Based on the aligned substantia nigra segmentations using a study-specific brain anatomical template, parametric maps of the probability of a voxel belonging to the substantia nigra were calculated for patients with various degrees of disease severity and controls. For each voxel in the substantia nigra, probability map of controls, correlations between signal-to-noise ratios on neuromelanin-sensitive MRI in patients with iRBD and Parkinson’s disease and clinical scores of motor disability, cognition and mood/behaviour were calculated. Our results showed that in patients, compared to the healthy control subjects, the volume of the substantia nigra was progressively reduced for increasing disease severity. The neuromelanin signal changes appeared to start in the posterolateral motor areas of the substantia nigra and then progressed to more medial areas of this region. The ratio between the volume of the substantia nigra in patients with Parkinson’s disease relative to the controls was best fitted by a mono-exponential decay. Based on this model, the pre-symptomatic phase of the disease started at 5.3 years before disease diagnosis, and 23.1% of the substantia nigra volume was lost at the time of diagnosis, which was in line with previous findings using post-mortem histology of the human substantia nigra and radiotracer studies of the human striatum. Voxel-wise patterns of correlation between neuromelanin-sensitive MRI signal-to-noise ratio and motor, cognitive and mood/behavioural clinical scores were localized in distinct regions of the substantia nigra. This localization reflected the functional organization of the nigrostriatal system observed in histological and electrophysiological studies in non-human primates (motor, cognitive and mood/behavioural domains). In conclusion, neuromelanin-sensitive MRI enabled us to assess voxel-wise modifications of substantia nigra’s morphology in vivo in humans, including healthy controls, patients with iRBD and patients with Parkinson’s disease, and identify their correlation with nigral function across all motor, cognitive and behavioural domains. This insight could help assess disease progression in drug trials of disease modification.

A high-resolution fMRI approach to characterize functionally distinct neural pathways within dopaminergic midbrain and nucleus accumbens during reward and salience processing

Processing of reward and salience without reward association are known to critically rely on the dopamine system. A growing body of evidence from animal studies suggests that both functions may be subserved by distinct subregions in midbrain and ventral striatum, specifically nucleus accumbens (NAcc). Yet in vivo investigation of these brain structures in humans has been rare. Here we examined blood oxygen level dependent signals in response to frequently presented rewarding events and infrequently presented neutral events in 20 healthy subjects using high-resolution functional magnetic resonance imaging (fMRI) for imaging the human midbrain and NAcc. The present findings revealed distinct activation patterns in brain regions of interest, namely increased activation in substantia nigra pars compacta (SNc) and dorsolateral NAcc in response to neutral events, while the VTA and both the ventromedial and dorsolateral NAcc were significantly activated due to rewarding events. Moreover, psychophysiological interaction analyses demonstrated regionally specialized processing pathways, such as a dorsolateral pathway when processing salience per se, i.e. increased functional interactions between SNc, dorsolateral NAcc and dorsolateral and medial prefrontal cortex (PFC); and a ventromedial pathway during reward processing, i.e. increased functional coupling between VTA and ventromedial NAcc. Thus, these findings may not only accelerate the integration of animal models of brain function with human neuroscience but may also improve diagnosis and treatment in patients with neuropsychiatric disorders such as schizophrenia and depression in which dopaminergic dysfunction and aberrant attribution of salience have been implicated.

Topographic distinction in long-term value signals between presumed dopamine neurons and presumed striatal projection neurons in behaving monkeys

Nigrostriatal dopamine (DA) projections are anatomically organized along the dorsolateral-ventromedial axis, conveying long-term value signals to the striatum for shaping actions toward multiple future rewards. The present study examines whether the topographic organization of long-term value signals are observed upon activity of presumed DA neurons and presumed striatal projection neurons (phasically active neurons, PANs), as predicted based on anatomical literature. Our results indicate that DA neurons in the dorsolateral midbrain encode long-term value signals on a short timescale, while ventromedial midbrain DA neurons encode such signals on a relatively longer timescale. Activity of the PANs in the dorsal striatum is more heterogeneous for encoding long-term values, although significant differences in long-term value signals were observed between the caudate nucleus and putamen. These findings suggest that topographic DA signals for long-term values are not simply transferred to striatal neurons, possibly due to the contribution of other projections to the striatum.

Physically active individuals look for more: An eye-tracking study of attentional bias

Attentional capture by exercise-related stimuli is important for the regulation of physical activity. Attentional processing underlying this capture has been investigated with indirect behavioral measures based on reaction times. To investigate more direct measures of visual spatial attention toward physical activity (vs. inactivity) stimuli, we used eye-tracking and a visual dot probe task in 77 young adults with various level of physical activity. Reaction times to detect a dot appearing in the area previously occupied by a physical activity (vs. inactivity) stimulus were an indirect measure of attentional bias. The first picture gaze and viewing time were more direct measures of attentional orienting and attentional engagement, respectively. Pupil dilation was an indicator of arousal. Reaction times revealed a two-way interaction between the location of the dot and participants' usual level of physical activity. Only participants with a high level of physical activity more quickly detected a dot when it appeared in the area previously occupied by a physical activity stimulus. Eye-tracking results showed greater odds of first gazing at physical activity stimuli and for a longer time, and a greater decrease in pupil size when viewing physical activity stimuli when usual level of physical activity was moderate or high, but not low. The variance explained in the outcomes ranged from 13.9% (pupil dilation) to 40% (reaction times). Overall, as hypothesized, compared to less physically active participants, participants who were more physically active demonstrated indirect (reaction times) and direct (first gaze, viewing time) evidence of a more pronounced attentional bias toward physical activity. Physical activity stimuli biased attention, with a pronounced effect when the level of physical activity was higher. These findings suggest that physical activity stimuli are relevant to the current concerns of moderately and highly active individuals.

Multiple associative structures created by reinforcement and incidental statistical learning mechanisms

Learning the structure of the world can be driven by reinforcement but also occurs incidentally through experience. Reinforcement learning theory has provided insight into how prediction errors drive updates in beliefs but less attention has been paid to the knowledge resulting from such learning. Here we contrast associative structures formed through reinforcement and experience of task statistics. BOLD neuroimaging in human volunteers demonstrates rigid representations of rewarded sequences in temporal pole and posterior orbito-frontal cortex, which are constructed backwards from reward. By contrast, medial prefrontal cortex and a hippocampal-amygdala border region carry reward-related knowledge but also flexible statistical knowledge of the currently relevant task model. Intriguingly, ventral striatum encodes prediction error responses but not the full RL- or statistically derived task knowledge. In summary, representations of task knowledge are derived via multiple learning processes operating at different time scales that are associated with partially overlapping and partially specialized anatomical regions.

Associative learning occurs through reinforcement mechanisms as well as incidentally through experience of statistical relationships. Here, the authors report that these two learning processes are associated with specialized anatomical regions that operate at different time scales.

Pupil dilation as an implicit measure of appetitive Pavlovian learning

Appetitive Pavlovian conditioning is a learning mechanism of fundamental biological and pathophysiological significance. Nonetheless, its exploration in humans remains sparse, which is partly attributed to the lack of an established psychophysiological parameter that aptly represents conditioned responding. This study evaluated pupil diameter and other ocular response measures (gaze dwelling time, blink duration and count) as indices of conditioning. Additionally, a learning model was used to infer participants' learning progress on the basis of their pupil dilation. Twenty-nine healthy volunteers completed an appetitive differential delay conditioning paradigm with a primary reward, while the ocular response measures along with other psychophysiological (heart rate, electrodermal activity, postauricular and eyeblink reflex) and behavioral (ratings, contingency awareness) parameters were obtained to examine the relation among different measures. A significantly stronger increase in pupil diameter, longer gaze duration and shorter eyeblink duration was observed in response to the reward-predicting cue compared to the control cue. The Pearce-Hall attention model best predicted the trial-by-trial pupil diameter. This conditioned response was corroborated by a pronounced heart rate deceleration to the reward-predicting cue, while no conditioning effect was observed in the electrodermal activity or startle responses. There was no discernible correlation between the psychophysiological response measures. These results highlight the potential value of ocular response measures as sensitive indices for representing appetitive conditioning.

Evidence for model-based encoding of Pavlovian contingencies in the human brain

Prominent accounts of Pavlovian conditioning successfully approximate the frequency and intensity of conditioned responses under the assumption that learning is exclusively model-free; that animals do not develop a cognitive map of events. However, these model-free approximations fall short of comprehensively capturing learning and behavior in Pavlovian conditioning. We therefore performed multivoxel pattern analysis of high-resolution functional MRI data in human participants to test for the encoding of stimulus-stimulus associations that could support model-based computations during Pavlovian conditioning. We found that dissociable sub-regions of the striatum encode predictions of stimulus-stimulus associations and predictive value, in a manner that is directly related to learning performance. Activity patterns in the orbitofrontal cortex were also found to be related to stimulus-stimulus as well as value encoding. These results suggest that the brain encodes model-based representations during Pavlovian conditioning, and that these representations are utilized in the service of behavior.

Behavioural evidence for parallel outcome-sensitive and outcome-insensitive Pavlovian learning systems in humans

There is a dichotomy in instrumental conditioning between goal-directed actions and habits that are distinguishable on the basis of their relative sensitivity to changes in outcome value. It is less clear whether a similar distinction applies in Pavlovian conditioning, where responses have been found to be predominantly outcome sensitive. To test for both devaluation insensitive and devaluation sensitive Pavlovian conditioning in humans, we conducted four experiments combining Pavlovian conditioning and outcome devaluation procedures while measuring multiple conditioned responses. Our results suggest that Pavlovian conditioning involves two distinct types of learning: one that learns the current value of the outcome which is sensitive to devaluation, and one that learns about the spatial localisation of the outcome which is insensitive to devaluation. Our findings have implications for the mechanistic understanding of Pavlovian conditioning and provide a more nuanced understanding of Pavlovian mechanisms that might contribute to a number of psychiatric disorders.

How does homeostasis happen? Integrative physiological, systems biological, and evolutionary perspectives

Homeostasis is a founding principle of integrative physiology. In current systems biology, however, homeostasis seems almost invisible. Is homeostasis a key goal driving body processes, or is it an emergent mechanistic fact? In this perspective piece, I propose that the integrative physiological and systems biological viewpoints about homeostasis reflect different epistemologies, different philosophies of knowledge. Integrative physiology is concept driven. It attempts to explain biological phenomena by continuous formation of theories that experimentation or observation can test. In integrative physiology, "function" refers to goals or purposes. Systems biology is data driven. It explains biological phenomena in terms of "omics"-i.e., genomics, gene expression, epigenomics, proteomics, and metabolomics-it depicts the data in computer models of complex cascades or networks, and it makes predictions from the models. In systems biology, "function" refers more to mechanisms than to goals. The integrative physiologist emphasizes homeostasis of internal variables such as Pco₂ and blood pressure. The systems biologist views these emphases as teleological and unparsimonious in that the "regulated variable" (e.g., arterial Pco₂ and blood pressure) and the "regulator" (e.g., the "carbistat" and "barostat") are unobservable constructs. The integrative physiologist views systems biological explanations as not really explanations but descriptions that cannot account for phenomena we humans believe exist, although they cannot be observed directly, such as feelings and, ultimately, the conscious mind. This essay reviews the history of the two epistemologies, emphasizing autonomic neuroscience. I predict rapprochement of integrative physiology with systems biology. The resolution will avoid teleological purposiveness, transcend pure mechanism, and incorporate adaptiveness in evolution, i.e., "Darwinian medicine."

Financial gain- and loss-related BOLD signals in the human ventral tegmental area and substantia nigra pars compacta

Neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNC) play central roles in reward-related behaviours. Nonhuman animal studies suggest that these neurons also process aversive events. However, our understanding of how the human VTA and SNC responds to such events is limited and has been hindered by the technical challenge of using functional magnetic resonance imaging (fMRI) to investigate a small structure where the signal is particularly vulnerable to physiological noise. Here we show, using methods optimized specifically for the midbrain (including high-resolution imaging, a novel registration protocol, and physiological noise modelling), a BOLD (blood-oxygen-level dependent) signal to both financial gain and loss in the VTA and SNC, along with a response to nil outcomes that are better or worse than expected in the VTA. Taken together, these findings suggest that the human VTA and SNC are involved in the processing of both appetitive and aversive financial outcomes in humans.

How pupil responses track value-based decision-making during and after reinforcement learning

Cognition can reveal itself in the pupil, as latent cognitive processes map onto specific pupil responses. For instance, the pupil dilates when we make decisions and these pupil size fluctuations reflect decision-making computations during and after a choice. Surprisingly little is known, however, about how pupil responses relate to decisions driven by the learned value of stimuli. This understanding is important, as most real-life decisions are guided by the outcomes of earlier choices. The goal of this study was to investigate which cognitive processes the pupil reflects during value-based decision-making. We used a reinforcement learning task to study pupil responses during value-based decisions and subsequent decision evaluations, employing computational modeling to quantitatively describe the underlying cognitive processes. We found that the pupil closely tracks reinforcement learning processes independently across participants and across trials. Prior to choice, the pupil dilated as a function of trial-by-trial fluctuations in value beliefs about the to-be chosen option and predicted an individual's tendency to exploit high value options. After feedback a biphasic pupil response was observed, the amplitude of which correlated with participants' learning rates. Furthermore, across trials, early feedback-related dilation scaled with value uncertainty, whereas later constriction scaled with signed reward prediction errors. These findings show that pupil size fluctuations can provide detailed information about the computations underlying value-based decisions and the subsequent updating of value beliefs. As these processes are affected in a host of psychiatric disorders, our results indicate that pupillometry can be used as an accessible tool to non-invasively study the processes underlying ongoing reinforcement learning in the clinic.

Human substantia nigra and ventral tegmental area involvement in computing social error signals during the ultimatum game

As models of shared expectations, social norms play an essential role in our societies. Since our social environment is changing constantly, our internal models of it also need to change. In humans, there is mounting evidence that neural structures such as the insula and the ventral striatum are involved in detecting norm violation and updating internal models. However, because of methodological challenges, little is known about the possible involvement of midbrain structures in detecting norm violation and updating internal models of our norms. Here, we used high-resolution cardiac-gated functional magnetic resonance imaging and a norm adaptation paradigm in healthy adults to investigate the role of the substantia nigra/ventral tegmental area (SN/VTA) complex in tracking signals related to norm violation that can be used to update internal norms. We show that the SN/VTA codes for the norm’s variance prediction error (PE) and norm PE with spatially distinct regions coding for negative and positive norm PE. These results point to a common role played by the SN/VTA complex in supporting both simple reward-based and social decision making.

Neural Habituation to Painful Stimuli Is Modulated by Dopamine: Evidence from a Pharmacological fMRI Study

In constantly changing environments, it is crucial to adaptively respond to threatening events. In particular, painful stimuli are not only processed in terms of their absolute intensity, but also with respect to their context. While contextual pain processing can simply entail the repeated processing of information (i.e., habituation), it can, in a more complex form, be expressed through predictions of magnitude before the delivery of nociceptive information (i.e., adaptive coding). Here, we investigated the brain regions involved in the adaptation to nociceptive electrical stimulation as well as their link to dopaminergic neurotransmission (placebo/haloperidol). The main finding is that haloperidol changed the habituation to the absolute pain intensity over time. More precisely, in the placebo condition, activity in left postcentral gyrus and midcingulate cortex increased linearly with pain intensity only in the beginning of the experiment and subsequently habituated. In contrast, when the dopaminergic system was blocked by haloperidol, a linear increase with pain intensity was present throughout the entire experiment. Finally, there were no adaptive coding effects in any brain regions. Together, our findings provide novel insights into the nature of pain processing by suggesting that dopaminergic neurotransmission plays a specific role for the habituation to painful stimuli over time.

Exploring Feature Dimensions to Learn a New Policy in an Uninformed Reinforcement Learning Task

When making a choice with limited information, we explore new features through trial-and-error to learn how they are related. However, few studies have investigated exploratory behaviour when information is limited. In this study, we address, at both the behavioural and neural level, how, when, and why humans explore new feature dimensions to learn a new policy for choosing a state-space. We designed a novel multi-dimensional reinforcement learning task to encourage participants to explore and learn new features, then used a reinforcement learning algorithm to model policy exploration and learning behaviour. Our results provide the first evidence that, when humans explore new feature dimensions, their values are transferred from the previous policy to the new online (active) policy, as opposed to being learned from scratch. We further demonstrated that exploration may be regulated by the level of cognitive ambiguity, and that this process might be controlled by the frontopolar cortex. This opens up new possibilities of further understanding how humans explore new features in an open-space with limited information.

Distinct prediction errors in mesostriatal circuits of the human brain mediate learning about the values of both states and actions: evidence from high-resolution fMRI

Prediction-error signals consistent with formal models of "reinforcement learning" (RL) have repeatedly been found within dopaminergic nuclei of the midbrain and dopaminoceptive areas of the striatum. However, the precise form of the RL algorithms implemented in the human brain is not yet well determined. Here, we created a novel paradigm optimized to dissociate the subtypes of reward-prediction errors that function as the key computational signatures of two distinct classes of RL models-namely, "actor/critic" models and action-value-learning models (e.g., the Q-learning model). The state-value-prediction error (SVPE), which is independent of actions, is a hallmark of the actor/critic architecture, whereas the action-value-prediction error (AVPE) is the distinguishing feature of action-value-learning algorithms. To test for the presence of these prediction-error signals in the brain, we scanned human participants with a high-resolution functional magnetic-resonance imaging (fMRI) protocol optimized to enable measurement of neural activity in the dopaminergic midbrain as well as the striatal areas to which it projects. In keeping with the actor/critic model, the SVPE signal was detected in the substantia nigra. The SVPE was also clearly present in both the ventral striatum and the dorsal striatum. However, alongside these purely state-value-based computations we also found evidence for AVPE signals throughout the striatum. These high-resolution fMRI findings suggest that model-free aspects of reward learning in humans can be explained algorithmically with RL in terms of an actor/critic mechanism operating in parallel with a system for more direct action-value learning.

Individual differences in eye blink rate predict both transient and tonic pupil responses during reversal learning

The pupil response under constant illumination can be used as a marker of cognitive processes. In the past, pupillary responses have been studied in the context of arousal and decision-making. However, recent work involving Parkinson's patients suggested that pupillary responses are additionally affected by reward sensitivity. Here, we build on these findings by examining how pupil responses are modulated by reward and loss while participants (N = 30) performed a Pavlovian reversal learning task. In fast (transient) pupil responses, we observed arousal-based influences on pupil size both during the expectation of upcoming value and the evaluation of unexpected monetary outcomes. Importantly, after incorporating eye blink rate (EBR), a behavioral correlate of striatal dopamine levels, we observed that participants with lower EBR showed stronger pupil dilation during the expectation of upcoming reward. Subsequently, when reward expectations were violated, participants with lower EBR showed stronger pupil responses after experiencing unexpected loss. Across trials, the detection of a reward contingency reversal was reflected in a slow (tonic) dilatory pupil response observed already several trials prior to the behavioral report. Interestingly, EBR correlated positively with this tonic detection response, suggesting that variability in the arousal-based detection response may reflect individual differences in striatal dopaminergic tone. Our results provide evidence that a behavioral marker of baseline striatal dopamine level (EBR) can potentially be used to describe the differential effects of value-based learning in the arousal-based pupil response.

Anatomical and functional organization of the human substantia nigra and its connections

We investigated the anatomical and functional organization of the human substantia nigra (SN) using diffusion and functional MRI data from the Human Connectome Project. We identified a tripartite connectivity-based parcellation of SN with a limbic, cognitive, motor arrangement. The medial SN connects with limbic striatal and cortical regions and encodes value (greater response to monetary wins than losses during fMRI), while the ventral SN connects with associative regions of cortex and striatum and encodes salience (equal response to wins and losses). The lateral SN connects with somatomotor regions of striatum and cortex and also encodes salience. Behavioral measures from delay discounting and flanker tasks supported a role for the value-coding medial SN network in decisional impulsivity, while the salience-coding ventral SN network was associated with motor impulsivity. In sum, there is anatomical and functional heterogeneity of human SN, which underpins value versus salience coding, and impulsive choice versus impulsive action.

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Optimal decision making mandates organisms learn the relevant features of choice options. Likewise, knowing how much effort we should expend can assume paramount importance. A mesolimbic network supports reward learning, but it is unclear whether other choice features, such as effort learning, rely on this same network. Using computational fMRI, we show parallel encoding of effort and reward prediction errors (PEs) within distinct brain regions, with effort PEs expressed in dorsomedial prefrontal cortex and reward PEs in ventral striatum. We show a common mesencephalic origin for these signals evident in overlapping, but spatially dissociable, dopaminergic midbrain regions expressing both types of PE. During action anticipation, reward and effort expectations were integrated in ventral striatum, consistent with a computation of an overall net benefit of a stimulus. Thus, we show that motivationally relevant stimulus features are learned in parallel dopaminergic pathways, with formation of an integrated utility signal at choice.

Beyond the Classic VTA: Extended Amygdala Projections to DA-Striatal Paths in the Primate

The central extended amygdala (CEA) has been conceptualized as a 'macrosystem' that regulates various stress-induced behaviors. Consistent with this, the CEA highly expresses corticotropin-releasing factor (CRF), an important modulator of stress responses. Stress alters goal-directed responses associated with striatal paths, including maladaptive responses such as drug seeking, social withdrawal, and compulsive behavior. CEA inputs to the midbrain dopamine (DA) system are positioned to influence striatal functions through mesolimbic DA-striatal pathways. However, the structure of this amygdala-CEA-DA neuron path to the striatum has been poorly characterized in primates. In primates, we combined neuronal tracer injections into various arms of the circuit through specific DA subpopulations to assess: (1) whether the circuit connecting amygdala, CEA, and DA cells follows CEA intrinsic organization, or a more direct topography involving bed nucleus vs central nucleus divisions; (2) CRF content of the CEA-DA path; and (3) striatal subregions specifically involved in CEA-DA-striatal loops. We found that the amygdala-CEA-DA path follows macrostructural subdivisions, with the majority of input/outputs converging in the medial central nucleus, the sublenticular extended amygdala, and the posterior lateral bed nucleus of the stria terminalis. The proportion of CRF+ outputs is >50%, and mainly targets the A10 parabrachial pigmented nucleus (PBP) and A8 (retrorubal field, RRF) neuronal subpopulations, with additional inputs to the dorsal A9 neurons. CRF-enriched CEA-DA projections are positioned to influence outputs to the 'limbic-associative' striatum, which is distinct from striatal regions targeted by DA cells lacking CEA input. We conclude that the concept of the CEA is supported on connectional grounds, and that CEA termination over the PBP and RRF neuronal populations can influence striatal circuits involved in associative learning.

Comparing functional MRI protocols for small, iron-rich basal ganglia nuclei such as the subthalamic nucleus at 7 T and 3 T

The basal ganglia (BG) form a network of subcortical nuclei. Functional magnetic resonance imaging (fMRI) in the BG could provide insight in its functioning and the underlying mechanisms of Deep Brain Stimulation (DBS). However, fMRI of the BG with high specificity is challenging, because the nuclei are small and variable in their anatomical location. High resolution fMRI at field strengths of 7 Tesla (T) could help resolve these challenges to some extent. A set of MR protocols was developed for functional imaging of the BG nuclei at 3 T and 7 T. The protocols were validated using a stop-signal reaction task (Logan et al. []: J Exp Psychol: Human Percept Perform 10:276-291). Compared with sub-millimeter 7 T fMRI protocols aimed at cortex, a reduction of echo time and spatial resolution was strictly necessary to obtain robust Blood Oxygen Level Dependent (BOLD) sensitivity in the BG. An fMRI protocol at 3 T with identical resolution to the 7 T showed no robust BOLD sensitivity in any of the BG nuclei. The results suggest that the subthalamic nucleus, as well as the substantia nigra, red nucleus, and the internal and external parts of the globus pallidus show increased activation in failed stop trials compared with successful stop and go trials. Hum Brain Mapp 38:3226-3248, 2017. © 2017 Wiley Periodicals, Inc.

Combining D-cycloserine with appetitive extinction learning modulates amygdala activity during recall

Appetitive Pavlovian conditioning plays a crucial role in the pathogenesis of drug addiction and conditioned reward cues can trigger craving and relapse even after long phases of abstinence. Promising preclinical work showed that the NMDA-receptor partial agonist D-cycloserine (DCS) facilitates Pavlovian extinction learning of fear and drug cues. Furthermore, DCS-augmented exposure therapy seems to be beneficial in various anxiety disorders, while the supposed working mechanism of DCS during human appetitive or aversive extinction learning is still not confirmed. To test the hypothesis that DCS administration before extinction training improves extinction learning, healthy adults (n=32) underwent conditioning, extinction, and extinction recall on three successive days in a randomized, double-blind, placebo-controlled fMRI design. Monetary wins and losses served as unconditioned stimuli during conditioning to probe appetitive and aversive learning. An oral dose of 50mg of DCS or placebo was administered 1h before extinction training and DCS effects during extinction recall were evaluated on a behavioral and neuronal level. We found attenuated amygdala activation in the DCS compared to the placebo group during recall of the extinguished appetitive cue, along with evidence for enhanced functional amygdala-vmPFC coupling in the DCS group. While the absence of additional physiological measures of conditioned responses during recall in this study prevent the evaluation of a behavioral DCS effect, our neuronal findings are in accordance with recent theories linking successful extinction recall in humans to modulatory top-down influences from the vmPFC that inhibit amygdala activation. Our results should encourage further translational studies concerning the usefulness of DCS to target maladaptive Pavlovian reward associations.

Behind binge eating: A review of food-specific adaptations of neurocognitive and neuroimaging tasks

Recurrent binge eating, or overeating accompanied by a sense of loss of control, is a major public health concern. Identifying similarities and differences among individuals with binge eating and those with other psychiatric symptoms and characterizing the deficits that uniquely predispose individuals to eating problems are essential to improving treatment. Research suggests that altered reward and control-related processes may contribute to dysregulated eating and other impulsive behaviors in binge-eating populations, but the best methods for reliably assessing the contributions of these processes to binge eating are unclear. In this review, we summarize standard neurocognitive and neuroimaging tasks that assess reward and control-related processes, describe adaptations of these tasks used to study eating and food-specific responsivity and deficits, and consider the advantages and limitations of these tasks. Future studies integrating both general and food-specific tasks with neuroimaging will improve understanding of the neurocognitive processes and neural circuits that contribute to binge eating and could inform novel interventions that more directly target or prevent this transdiagnostic behavior.

Partial lesion of dopamine neurons of rat substantia nigra impairs conditioned place aversion but spares conditioned place preference

Midbrain dopamine neurons play critical roles in reward- and aversion-driven associative learning. However, it is not clear whether they do this by a common mechanism or by separate mechanisms that can be dissociated. In the present study we addressed this question by testing whether a partial lesion of the dopamine neurons of the rat SNc has comparable effects on conditioned place preference (CPP) learning and conditioned place aversion (CPA) learning. Partial lesions of dopamine neurons in the rat substantia nigra pars compacta (SNc) induced by bilateral intranigral infusion of 6-hydroxydopamine (6-OHDA, 3μg/side) or 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP, 200μg/side) impaired learning of conditioned place aversion (CPA) without affecting conditioned place preference (CPP) learning. Control experiments demonstrated that these lesions did not impair motor performance and did not alter the hedonic value of the sucrose and quinine. The number of dopamine neurons in the caudal part of the SNc positively correlated with the CPP scores of the 6-OHDA rats and negatively correlated with CPA scores of the SHAM rats. In addition, the CPA scores of the 6-OHDA rats positively correlated with the tissue content of striatal dopamine. Insomuch as reward-driven learning depends on an increase in dopamine release by nigral neurons, these findings show that this mechanism is functional even in rats with a partial lesion of the SNc. On the other hand, if aversion-driven learning depends on a reduction of extracellular dopamine in the striatum, the present study suggests that this mechanism is no longer functional after the partial SNc lesion.

Modeling Trait Anxiety: From Computational Processes to Personality

Computational methods are increasingly being applied to the study of psychiatric disorders. Often, this involves fitting models to the behavior of individuals with subclinical character traits that are known vulnerability factors for the development of psychiatric conditions. Anxiety disorders can be examined with reference to the behavior of individuals high in "trait" anxiety, which is a known vulnerability factor for the development of anxiety and mood disorders. However, it is not clear how this self-report measure relates to neural and behavioral processes captured by computational models. This paper reviews emerging computational approaches to the study of trait anxiety, specifying how interacting processes susceptible to analysis using computational models could drive a tendency to experience frequent anxious states and promote vulnerability to the development of clinical disorders. Existing computational studies are described in the light of this perspective and appropriate targets for future studies are discussed.

Learning, Reward, and Decision Making

In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward-related behavior, including a dichotomy between the goal-directed or model-based system and the habitual or model-free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditioning. We evaluate evidence from neuroscience supporting the existence of at least partly distinct neuronal substrates contributing to the key computations necessary for the function of these different control systems. We consider the nature of the interactions between these systems and show how these interactions can lead to either adaptive or maladaptive behavioral outcomes. We then review evidence that an additional system guides inference concerning the hidden states of other agents, such as their beliefs, preferences, and intentions, in a social context. We also describe emerging evidence for an arbitration mechanism between model-based and model-free reinforcement learning, placing such a mechanism within the broader context of the hierarchical control of behavior.

Glutamate neurons are intermixed with midbrain dopamine neurons in nonhuman primates and humans

The rodent ventral tegmental area (VTA) and substantia nigra pars compacta (SNC) contain dopamine neurons intermixed with glutamate neurons (expressing vesicular glutamate transporter 2; VGluT2), which play roles in reward and aversion. However, identifying the neuronal compositions of the VTA and SNC in higher mammals has remained challenging. Here, we revealed VGluT2 neurons within the VTA and SNC of nonhuman primates and humans by simultaneous detection of VGluT2 mRNA and tyrosine hydroxylase (TH; for identification of dopamine neurons). We found that several VTA subdivisions share similar cellular compositions in nonhuman primates and humans; their rostral linear nuclei have a high prevalence of VGluT2 neurons lacking TH; their paranigral and parabrachial pigmented nuclei have mostly TH neurons, and their parabrachial pigmented nuclei have dual VGluT2-TH neurons. Within nonhuman primates and humans SNC, the vast majority of neurons are TH neurons but VGluT2 neurons were detected in the pars lateralis subdivision. The demonstration that midbrain dopamine neurons are intermixed with glutamate or glutamate-dopamine neurons from rodents to humans offers new opportunities for translational studies towards analyzing the roles that each of these neurons play in human behavior and in midbrain-associated illnesses such as addiction, depression, schizophrenia, and Parkinson's disease.