Complementary neural correlates of motivation in dopaminergic and noradrenergic neurons of monkeys

Spatial localization of hippocampal replay requires dopamine signaling

Sequenced reactivations of hippocampal neurons called replays, concomitant with sharp-wave ripples in the local field potential, are critical for the consolidation of episodic memory, but whether replays depend on the brain's reward or novelty signals is unknown. Here we combined chemogenetic silencing of dopamine neurons in ventral tegmental area (VTA) and simultaneous electrophysiological recordings in dorsal hippocampal CA1, in freely behaving rats experiencing changes to reward magnitude and environmental novelty. Surprisingly, VTA silencing did not prevent ripple increases where reward was increased, but caused dramatic, aberrant ripple increases where reward was unchanged. These increases were associated with increased reverse-ordered replays. On familiar tracks this effect disappeared, and ripples tracked reward prediction error, indicating that non-VTA reward signals were sufficient to direct replay. Our results reveal a novel dependence of hippocampal replay on dopamine, and a role for a VTA-independent reward prediction error signal that is reliable only in familiar environments.

Pupillary Responses Reflect Dynamic Changes in Multiple Cognitive Factors During Associative Learning in Primates

Associative learning involves complex interactions of multiple cognitive factors. While adult subjects can articulate these factors verbally, for model animals such as macaques, we rely on behavioral outputs. In our study, we used pupillary responses as an alternative measure to capture these underlying cognitive changes. We recorded the dynamic changes in the pupils of three male macaques when they learned the associations between visual stimuli and reward sizes under the classical Pavlovian experimental paradigm. We found that during the long-term learning process, the gradual changes in the pupillary response reflect the changes in the cognitive state of the animals. The pupillary response can be explained by a linear combination of components corresponding to multiple cognitive factors. These components reflect the impact of visual stimuli on the pupils, the prediction of reward values associated with the visual stimuli, and the macaques' understanding of the current experimental reward rules. The changing patterns of these factors during interday and intraday learning clearly demonstrate the enhancement of current reward-stimulus association and the weakening of previous reward-stimulus association. Our study shows that the dynamic response of pupils can serve as an objective indicator to characterize the psychological changes of animals, understand their learning process, and provide important tools for exploring animal behavior during the learning process.

Alpha1-adrenergic receptor blockade in the ventral tegmental area attenuates acquisition of cocaine-induced pavlovian associative learning

Activity of the alpha₁-adrenergic receptor (α₁-AR) in the ventral tegmental area (VTA) modulates dopaminergic activity, implying its modulatory role in the behavioral functions of the dopamine (DA) system. Indeed, intra-VTA α₁-AR blockade attenuates conditioned stimulus dependent behaviors such as drug seeking responses signifying a role of the noradrenergic signaling in the VTA in conditioned behaviors. Importantly, the role of the VTA α₁-AR activity in Pavlovian associative learning with positive outcomes remains unknown. Here, we aimed to examine how intra-VTA α₁-AR blockade affects acquisition of cocaine-induced Pavlovian associative learning in the conditioned place preference (CPP) paradigm. The impact of α₁-AR blockade on cocaine-reinforced operant responding and cocaine-evoked ultrasonic vocalizations (USVs) was also studied. In addition, both α₁-AR immunoreactivity in the VTA and its role in phasic DA release in the nucleus accumbens (NAc) were assessed. We demonstrated cellular localization of α₁-AR expression in the VTA, providing a neuroanatomical substrate for the α₁-AR mechanism. We showed that prazosin (α₁-AR selective antagonist; 1 μg/0.5 μl) microinfusion attenuated electrically evoked DA transients in the NAc and dose-dependently (0.1-1 μg/0.5 μl) prevented the acquisition of cocaine CPP but did not affect cocaine-reinforced operant responding nor cocaine-induced positive affective state (measured as USVs). We propose that the VTA α₁-AR signaling is necessary for the acquisition of Pavlovian associative learning but does not encode hedonic value. Thus, α₁-AR signaling in the VTA might underlie salience encoding of environmental stimuli and reflect an ability of alerting/orienting functions, originating from bottom-up information processing to guide behaviors.

Dopaminergic and noradrenergic modulation of stress-induced alterations in brain activation associated with goal-directed behaviour

BACKGROUND

Acute stress is thought to reduce goal-directed behaviour, an effect purportedly associated with stress-induced release of catecholamines. In contrast, experimentally increased systemic catecholamine levels have been shown to increase goal-directed behaviour. Whether experimentally increased catecholamine function can modulate stress-induced reductions in goal-directed behaviour and its neural substrates, is currently unknown.

AIM

To assess whether and how experimentally induced increases in dopamine and noradrenaline contribute to the acute stress effects on goal-directed behaviour and associated brain activation.

METHODS

One hundred participants underwent a stress induction protocol (Maastricht acute stress test; MAST) or a control procedure and received methylphenidate (MPH) (40 mg, oral) or placebo according to a 2 × 2 between-subjects design. In a well-established instrumental learning paradigm, participants learnt stimulus-response-outcome associations, after which rewards were selectively devalued. Participants' brain activation and associated goal-directed behaviour were assessed in a magnetic resonance imaging scanner at peak cortisol/MPH concentrations.

RESULTS

The MAST and MPH increased physiological measures of stress (salivary cortisol and blood pressure), but only MAST increased subjective measures of stress. MPH modulated stress effects on activation of brain areas associated with goal-directed behaviour, including insula, putamen, amygdala, medial prefrontal cortex, frontal pole and orbitofrontal cortex. However, MPH did not modulate the tendency of stress to induce a reduction in goal-directed behaviour.

CONCLUSION

Our neuroimaging data suggest that MPH-induced increases in dopamine and noradrenaline reverse stress-induced changes in key brain regions associated with goal-directed behaviour, while behavioural effects were absent. These effects may be relevant for preventing stress-induced maladaptive behaviour like in addiction or binge eating disorder.

The acute exercise-cognition interaction: From the catecholamines hypothesis to an interoception model

An interoception model for the acute exercise-cognition interaction is presented. During exercise following the norepinephrine threshold, interoceptive feedback induces increased tonic release of extracellular catecholamines, facilitating phasic release hence better cognitive performance of executive functions. When exercise intensity increases to maximum, the nature of task-induced norepinephrine release from the locus coeruleus is dependent on interaction between motivation, perceived effort costs and perceived availability of resources. This is controlled by interaction between the rostral and dorsolateral prefrontal cortices, orbitofrontal cortex, anterior cingulate cortex and anterior insula cortex. If perceived available resources are sufficient to meet predicted effort costs and reward value is high, tonic release from the locus coeruleus is attenuated thus facilitating phasic release, therefore cognition is not inhibited. However, if perceived available resources are insufficient to meet predicted effort costs or reward value is low, tonic release from the locus coeruleus is induced, attenuating phasic release. As a result, cognition is inhibited, although long-term memory and tasks that require switching to new stimuli-response couplings are probably facilitated.

The unconscious mental inhibiting process of human maximal voluntary contraction

Human maximal voluntary contraction (MVC) is believed to be limited by neural inhibition. Motivational goal priming alters background states of the motor system, leading to enhanced MVC. However, the mechanisms that determine the constant inhibition of force exertion in the motor system remain unclear. The primary behavioural goal of MVC is maximal voluntary force exertion. The final expected or desired state of this behavioural goal is explicitly demonstrated with words related to physical exertion, such as 'maximal', irrespective of the possibility of demand-like properties in participants' minds, such as attainability and/or desirability of the goal. For the primed maximal goal state, most trial results fail to meet expectations, demonstrating negative affect that, without awareness, contributes to the mentioned inhibitory mechanism underlying MVC. We therefore speculated that the behavioural goal of MVC contributes to neural inhibitory mechanisms underlying MVC. In our study, we used a previously developed paradigm (Takarada and Nozaki in Scientific Reports 8: 10135, 2018) in which subliminal visual priming stimuli such as the physical exertion-related words "perform" and "exert" were presented to 12 healthy participants and were followed by supraliminal words that were the word "maximal" or neutral.We found that when combined with the term 'maximal' in the consciously visible form, the effect of this subliminal motor-goal priming in inducing pupil dilation and stronger action preparation/execution was abolished without conscious awareness. This is the first objective evidence of motor inhibitory effect-predicting patterns of pupil-linked noradrenergic activity as a signature of a type of mental inhibition underlying the MVC behavioural goal.

Expectation-driven novelty effects in episodic memory

Novel and unexpected stimuli are often prioritised in memory, given their inherent salience. Nevertheless, not all forms of novelty show such an enhancement effect. Here, we discuss the role expectation plays in modulating the way novelty affects memory processes, circuits, and subsequent performance. We first review independent effects of expectation on memory, and then consider how different types of novelty are characterised by expectation. We argue that different types of novelty defined by expectation implicate differential neurotransmission in memory formation brain regions and may also result in the creation of different types of memory. Contextual novelty, which is unexpected by definition, is often associated with better recollection, supported by dopaminergic-hippocampal interactions. On the other hand, expected stimulus novelty is supported by engagement of medial temporal cortices, as well as the hippocampus, through cholinergic modulation. Furthermore, when expected stimulus novelty results in enhanced memory, it is predominantly driven by familiarity. The literature reviewed here highlights the complexity of novelty-sensitive memory systems, the distinction between types of novelty, and how they are differentially affected by expectancy.

Dose-response effects of d-amphetamine on effort-based decision-making and reinforcement learning

Effort-related decision-making and reward learning are both dopamine-dependent, but preclinical research suggests they depend on different dopamine signaling dynamics. Therefore, the same dose of a dopaminergic medication could have differential effects on effort for reward vs. reward learning. However, no study has tested how effort and reward learning respond to the same dopaminergic medication within subjects. The current study aimed to test the effect of therapeutic doses of d-amphetamine on effort for reward and reward learning in the same healthy volunteers. Participants (n = 30) completed the Effort Expenditure for Reward Task (EEfRT) measure of effort-related decision-making, and the Probabilistic Reward Task (PRT) measure of reward learning, under placebo and two doses of d-amphetamine (10 mg, and 20 mg). Secondarily, we examined whether the individual characteristics of baseline working memory and willingness to exert effort for reward moderated the effects of d-amphetamine. d-Amphetamine increased willingness to exert effort, particularly at low to intermediate expected values of reward. Computational modeling analyses suggested this was due to decreased effort discounting rather than probability discounting or decision consistency. Both baseline effort and working memory emerged as moderators of this effect, such that d-amphetamine increased effort more in individuals with lower working memory and lower baseline effort, also primarily at low to intermediate expected values of reward. In contrast, d-amphetamine had no significant effect on reward learning. These results have implications for treatment of neuropsychiatric disorders, which may be characterized by multiple underlying reward dysfunctions.

Pharmacological evidence for the implication of noradrenaline in effort

The trade-off between effort and reward is one of the main determinants of behavior, and its alteration is at the heart of major disorders such as depression or Parkinson's disease. Monoaminergic neuromodulators are thought to play a key role in this trade-off, but their relative contribution remains unclear. Rhesus monkeys (Macaca mulatta) performed a choice task requiring a trade-off between the volume of fluid reward and the amount of force to be exerted on a grip. In line with a causal role of noradrenaline in effort, decreasing noradrenaline levels with systemic clonidine injections (0.01 mg/kg) decreased exerted force and enhanced the weight of upcoming force on choices, without any effect on reward sensitivity. Using computational modeling, we showed that a single variable ("effort") could capture the amount of resources necessary for action and control both choices (as a variable for decision) and force production (as a driving force). Critically, the multiple effects of noradrenaline manipulation on behavior could be captured by a specific modulation of this single variable. Thus, our data strongly support noradrenaline's implication in effort processing.

The effects of cognitive abilities and task demands on tonic and phasic pupil sizes

Previous studies on individual differences in pupil size of healthy individuals and their relation to performance have been inconclusive. Using a novel approach, we tested the effect of general cognitive abilities and level of task performance on pretrial baseline and task-evoked pupil (TEP) sizes (N = 116) while we manipulated the level of task demands using a multiple object tracking task. Results did not reveal an effect of general cognitive abilities, estimated by working memory capacity and gF scores, on either baseline or TEP sizes. In contrast, we found an interaction in TEP sizes between level of overall MOT performance and task demands. We propose that individual differences in TEP sizes are related to state-specific level of task performance and task demands, probably in combination with other factors like age, personality traits, and state-specific level of motivation and arousal. We also suggest methodological confounds that may cause the previous inconclusive findings.

The role of norepinephrine in the pathophysiology of schizophrenia

Several lines of evidence have suggested for decades a role for norepinephrine (NE) in the pathophysiology and treatment of schizophrenia. Recent experimental findings reveal anatomical and physiological properties of the locus coeruleus-norepinephrine (LC-NE) system and its involvement in brain function and cognition. Here, we integrate these two lines of evidence. First, we review the functional and structural properties of the LC-NE system and its impact on functional brain networks, cognition, and stress, with special emphasis on recent experimental and theoretical advances. Subsequently, we present an update about the role of LC-associated functions for the pathophysiology of schizophrenia, focusing on the cognitive and motivational deficits. We propose that schizophrenia phenomenology, in particular cognitive symptoms, may be explained by an abnormal interaction between genetic susceptibility and stress-initiated LC-NE dysfunction. This in turn, leads to imbalance between LC activity modes, dysfunctional regulation of brain network integration and neural gain, and deficits in cognitive functions. Finally, we suggest how recent development of experimental approaches can be used to characterize LC function in schizophrenia.

Time-resolved neurotransmitter detection in mouse brain tissue using an artificial intelligence-nanogap

The analysis of neurotransmitters in the brain helps to understand brain functions and diagnose Parkinson’s disease. Pharmacological inhibition experiments, electrophysiological measurement of action potentials, and mass analysers have been applied for this purpose; however, these techniques do not allow direct neurotransmitter detection with good temporal resolution by using nanometre-sized electrodes. Hence, we developed a method for direct observation of a single neurotransmitter molecule with a gap width of ≤ 1 nm and on the millisecond time scale. It consists of measuring the tunnelling current that flows through a single-molecule by using nanogap electrodes and machine learning analysis. Using this method, we identified dopamine, serotonin, and norepinephrine neurotransmitters with high accuracy at the single-molecule level. The analysis of the mouse striatum and cerebral cortex revealed the order of concentration of the three neurotransmitters. Our method will be developed to investigate the neurotransmitter distribution in the brain with good temporal resolution.

Noradrenergic But Not Dopaminergic Neurons Signal Task State Changes and Predict Reengagement After a Failure

The two catecholamines, noradrenaline and dopamine, have been shown to play comparable roles in behavior. Both noradrenergic and dopaminergic neurons respond to cues predicting reward availability and novelty. However, even though both are thought to be involved in motivating actions, their roles in motivation have seldom been directly compared. We therefore examined the activity of putative noradrenergic neurons in the locus coeruleus and putative midbrain dopaminergic neurons in monkeys cued to perform effortful actions for rewards. The activity in both regions correlated with engagement with a presented option. By contrast, only noradrenaline neurons were also (i) predictive of engagement in a subsequent trial following a failure to engage and (ii) more strongly activated in nonrepeated trials, when cues indicated a new task condition. This suggests that while both catecholaminergic neurons are involved in promoting action, noradrenergic neurons are sensitive to task state changes, and their influence on behavior extends beyond the immediately rewarded action.

A neural network for information seeking

Humans and other animals often show a strong desire to know the uncertain rewards their future has in store, even when they cannot use this information to influence the outcome. However, it is unknown how the brain predicts opportunities to gain information and motivates this information-seeking behavior. Here we show that neurons in a network of interconnected subregions of primate anterior cingulate cortex and basal ganglia predict the moment of gaining information about uncertain rewards. Spontaneous increases in their information prediction signals are followed by gaze shifts toward objects associated with resolving uncertainty, and pharmacologically disrupting this network reduces the motivation to seek information. These findings demonstrate a cortico-basal ganglia mechanism responsible for motivating actions to resolve uncertainty by seeking knowledge about the future.

Animals resolve uncertainty by seeking knowledge about the future. How the brain controls this is unclear. The authors show that a network including primate anterior cingulate cortex and basal ganglia encodes opportunities to gain information about uncertain rewards and mediates information seeking.

Modulation of locus coeruleus activity by novel oddball stimuli

It has long been known from animal literature that the locus coeruleus (LC), the source region of noradrenergic neurons in the brain, is sensitive to unexpected, novel, and other salient events. In humans, however, direct assessment of LC activity has proven to be challenging due to its small size and difficult localization, which is why noradrenergic activity has often been assessed using more indirect measures such as electroencephalography (EEG) and pupil recordings. Here, we combined high-resolution functional magnetic resonance imaging (fMRI) with a special anatomical sequence to assess neural activity in the LC in response to different types of salient stimuli in an oddball paradigm (novel neutral oddballs, novel emotional oddballs, and familiar target oddballs). We found a significant linear increase of LC activity from standard trials, over familiar target oddballs, to novel neutral and novel emotional oddballs. Importantly, when breaking down this linear trend, only novel oddball stimuli led to robust activity increases as compared to standard trials, with no statistical difference between neutral and emotional ones. This pattern suggests that activity modulations in the LC in the present study were mainly driven by stimulus novelty, rather than by emotional saliency, task relevance, or contextual novelty alone. Moreover, the absence of significant activity modulations in response to target oddballs (which were reported in a recent study) suggests that the LC represents relative rather than absolute saliency of a stimulus in its respective context.

How pupil responses track value-based decision-making during and after reinforcement learning

Cognition can reveal itself in the pupil, as latent cognitive processes map onto specific pupil responses. For instance, the pupil dilates when we make decisions and these pupil size fluctuations reflect decision-making computations during and after a choice. Surprisingly little is known, however, about how pupil responses relate to decisions driven by the learned value of stimuli. This understanding is important, as most real-life decisions are guided by the outcomes of earlier choices. The goal of this study was to investigate which cognitive processes the pupil reflects during value-based decision-making. We used a reinforcement learning task to study pupil responses during value-based decisions and subsequent decision evaluations, employing computational modeling to quantitatively describe the underlying cognitive processes. We found that the pupil closely tracks reinforcement learning processes independently across participants and across trials. Prior to choice, the pupil dilated as a function of trial-by-trial fluctuations in value beliefs about the to-be chosen option and predicted an individual's tendency to exploit high value options. After feedback a biphasic pupil response was observed, the amplitude of which correlated with participants' learning rates. Furthermore, across trials, early feedback-related dilation scaled with value uncertainty, whereas later constriction scaled with signed reward prediction errors. These findings show that pupil size fluctuations can provide detailed information about the computations underlying value-based decisions and the subsequent updating of value beliefs. As these processes are affected in a host of psychiatric disorders, our results indicate that pupillometry can be used as an accessible tool to non-invasively study the processes underlying ongoing reinforcement learning in the clinic.

What Is the Relationship between Dopamine and Effort?

The trade-off between reward and effort is at the heart of most behavioral theories, from ecology to economics. Compared to reward, however, effort remains poorly understood, both at the behavioral and neurophysiological levels. This is important because unwillingness to overcome effort to gain reward is a common feature of many neuropsychiatric and neurological disorders. A recent surge in interest in the neurobiological basis of effort has led to seemingly conflicting results regarding the role of dopamine. We argue here that, upon closer examination, there is actually striking consensus across studies: dopamine primarily codes for future reward but is less sensitive to anticipated effort cost. This strong association between dopamine and the incentive effects of rewards places dopamine in a key position to promote reward-directed action.

How do memory systems detect and respond to novelty?

The efficiency of the memory system lies not only in its readiness to detect and retrieve old stimuli but also in its ability to detect and integrate novel information. In this review, we discuss recent evidence suggesting that the neural substrates sensitive to detecting familiarity and novelty are not entirely overlapping. Instead, these partially distinct familiarity and novelty signals are integrated to support recognition memory decisions. We propose here that the mediodorsal thalamus is critical for familiarity detection, and for combining novelty signals from the medial temporal lobe cortex with the relative familiarity outputs of computations performed in other cortical structures, especially the prefrontal cortex. Importantly, we argue that the anterior hippocampus has a prominent role in detecting novelty and in communicating this with midbrain and striatal structures. We argue that different types of novelty (absolute or contextual) engage different neurotransmitter systems that converge in the hippocampus. We suggest that contextual or unexpected novelty triggers dopaminergic hippocampal-midbrain coupling and noradrenergic-mediated pupil dilation. In contrast, absolute novelty triggers cholinergic-mediated hippocampal encoding accompanied by diminished pupil dilation. These two, distinct hippocampal encoding mechanisms both lead to later recollection but are sensitive to different types of novelty. We conclude that this neurotransmitter-mediated hippocampal encoding establishes the hippocampus in an encoding mode that briefly prevents the engagement of retrieval.

Locus Coeruleus Activity Strengthens Prioritized Memories Under Arousal

Recent models posit that bursts of locus ceruleus (LC) activity amplify neural gain such that limited attention and encoding resources focus even more on prioritized mental representations under arousal. Here, we tested this hypothesis in human males and females using fMRI, neuromelanin MRI, and pupil dilation, a biomarker of arousal and LC activity. During scanning, participants performed a monetary incentive encoding task in which threat of punishment motivated them to prioritize encoding of scene images over superimposed objects. Threat of punishment elicited arousal and selectively enhanced memory for goal-relevant scenes. Furthermore, trial-level pupil dilations predicted better scene memory under threat, but were not related to object memory outcomes. fMRI analyses revealed that greater threat-evoked pupil dilations were positively associated with greater scene encoding activity in LC and parahippocampal cortex, a region specialized to process scene information. Across participants, this pattern of LC engagement for goal-relevant encoding was correlated with neuromelanin signal intensity, providing the first evidence that LC structure relates to its activation pattern during cognitive processing. Threat also reduced dynamic functional connectivity between high-priority (parahippocampal place area) and lower-priority (lateral occipital cortex) category-selective visual cortex in ways that predicted increased memory selectivity. Together, these findings support the idea that, under arousal, LC activity selectively strengthens prioritized memory representations by modulating local and functional network-level patterns of information processing.SIGNIFICANCE STATEMENT Adaptive behavior relies on the ability to select and store important information amid distraction. Prioritizing encoding of task-relevant inputs is especially critical in threatening or arousing situations, when forming these memories is essential for avoiding danger in the future. However, little is known about the arousal mechanisms that support such memory selectivity. Using fMRI, neuromelanin MRI, and pupil measures, we demonstrate that locus ceruleus (LC) activity amplifies neural gain such that limited encoding resources focus even more on prioritized mental representations under arousal. For the first time, we also show that LC structure relates to its involvement in threat-related encoding processes. These results shed new light on the brain mechanisms by which we process important information when it is most needed.

Genesis and Maintenance of Attentional Biases: The Role of the Locus Coeruleus-Noradrenaline System

Emotionally arousing events are typically better remembered than mundane ones, in part because emotionally relevant aspects of our environment are prioritized in attention. Such biased attentional tuning is itself the result of associative processes through which we learn affective and motivational relevance of cues. We propose that the locus coeruleus-noradrenaline (LC-NA) system plays an important role in the genesis of attentional biases through associative learning processes as well as their maintenance. We further propose that individual differences in and disruptions of the LC-NA system underlie the development of maladaptive biases linked to psychopathology. We provide support for the proposed role of the LC-NA system by first reviewing work on attentional biases in development and its link to psychopathology in relation to alterations and individual differences in NA availability. We focus on pharmacological manipulations to demonstrate the effect of a disrupted system as well as the ADRA2b polymorphism as a tool to investigate naturally occurring differences in NA availability. We next review associative learning processes that-modulated by the LC-NA system-result in such implicit attentional biases. Further, we demonstrate how NA may influence aversive and appetitive conditioning linked to anxiety disorders as well as addiction and depression.

Noradrenaline blockade specifically enhances metacognitive performance

Impairments in metacognition, the ability to accurately report one’s performance, are common in patients with psychiatric disorders, where a putative neuromodulatory dysregulation provides the rationale for pharmacological interventions. Previously, we have shown how unexpected arousal modulates metacognition (Allen et al., 2016). Here, we report a double-blind, placebo-controlled, study that examined specific effects of noradrenaline and dopamine on both metacognition and perceptual decision making. Signal theoretic analysis of a global motion discrimination task with adaptive performance staircasing revealed that noradrenergic blockade (40 mg propranolol) significantly increased metacognitive performance (type-II area under the curve, AUROC2), but had no impact on perceptual decision making performance. Blockade of dopamine D2/3 receptors (400 mg amisulpride) had no effect on either metacognition or perceptual decision making. Our study is the first to show a pharmacological enhancement of metacognitive performance, in the absence of any effect on perceptual decision making. This enhancement points to a regulatory role for noradrenergic neurotransmission in perceptual metacognition.

DOI:http://dx.doi.org/10.7554/eLife.24901.001

Midbrain Dopamine Neurons Signal Belief in Choice Accuracy during a Perceptual Decision

Central to the organization of behavior is the ability to predict the values of outcomes to guide choices. The accuracy of such predictions is honed by a teaching signal that indicates how incorrect a prediction was ("reward prediction error," RPE). In several reinforcement learning contexts, such as Pavlovian conditioning and decisions guided by reward history, this RPE signal is provided by midbrain dopamine neurons. In many situations, however, the stimuli predictive of outcomes are perceptually ambiguous. Perceptual uncertainty is known to influence choices, but it has been unclear whether or how dopamine neurons factor it into their teaching signal. To cope with uncertainty, we extended a reinforcement learning model with a belief state about the perceptually ambiguous stimulus; this model generates an estimate of the probability of choice correctness, termed decision confidence. We show that dopamine responses in monkeys performing a perceptually ambiguous decision task comply with the model's predictions. Consequently, dopamine responses did not simply reflect a stimulus' average expected reward value but were predictive of the trial-to-trial fluctuations in perceptual accuracy. These confidence-dependent dopamine responses emerged prior to monkeys' choice initiation, raising the possibility that dopamine impacts impending decisions, in addition to encoding a post-decision teaching signal. Finally, by manipulating reward size, we found that dopamine neurons reflect both the upcoming reward size and the confidence in achieving it. Together, our results show that dopamine responses convey teaching signals that are also appropriate for perceptual decisions.

Effects of a Neonatal Experience Involving Reward Through Maternal Contact on the Noradrenergic System of the Rat Prefrontal Cortex

The noradrenergic system plays an important role in prefrontal cortex (PFC) function. Since early life experiences play a crucial role in programming brain function, we investigated the effects of a neonatal experience involving reward through maternal contact on the noradrenergic system of the rat PFC. Rat pups were exposed during Postnatal days (PNDs) 10-13, to a T-maze in which contact with the mother was used as a reward (RER). RER males had higher norepinephrine levels in the PFC both on PND 13 and in adulthood. The RER experience resulted in adulthood in increased levels of the active demethylase GADD45b, hypomethylation of the β1 adrenergic receptor (ADRB1) gene promoter, and consequent enhanced expression of its mRNA in the PFC. In addition, protein and binding levels of the ADRB1, as well as those of its downstream effector phosphorylated cAMP response element-binding protein were elevated in RER males. The higher activity of the PFC noradrenergic system of the RER males was reflected in their superior performance in the olfactory discrimination and the contextual fear extinction, 2 PFC noradrenergic system-dependent behavioral tasks.

Noradrenaline and dopamine neurons in the reward/effort trade-off: a direct electrophysiological comparison in behaving monkeys

Motivation determines multiple aspects of behavior, including action selection and energization of behavior. Several components of the underlying neural systems have been examined closely, but the specific role of the different neuromodulatory systems in motivation remains unclear. Here, we compare directly the activity of dopaminergic neurons from the substantia nigra pars compacta and noradrenergic neurons from the locus coeruleus in monkeys performing a task manipulating the reward/effort trade-off. Consistent with previous reports, dopaminergic neurons encoded the expected reward, but we found that they also anticipated the upcoming effort cost in connection with its negative influence on action selection. Conversely, the firing of noradrenergic neurons increased with both pupil dilation and effort production in relation to the energization of behavior. Therefore, this work underlines the contribution of dopamine to effort-based decision making and uncovers a specific role of noradrenaline in energizing behavior to face challenges.

Noradrenaline and dopamine: sharing the Workload

Varazzani and colleagues explored how noradrenergic and dopaminergic neural activity relates to cost/benefit decisions involving effort. They show these systems may play complementary roles in resolving these decisions; dopamine encodes cost-discounted values of rewards, whereas noradrenergic cells modify activity in relation to the amount of effort required to obtain them.

Modulating the map: dopaminergic tuning of hippocampal spatial coding and interactions

Salient events activate the midbrain dopaminergic system and have important impacts on various aspects of mnemonic function, including the stability of hippocampus-dependent memories. Dopamine is also central to modulation of neocortical memory processing, particularly during prefrontal cortex-dependent working memory. Here, we review the current state of the circuitry and physiology underlying dopamine's actions, suggesting that--alongside local effects within hippocampus and prefrontal cortex--dopamine released from the midbrain ventral tegmental area is well positioned to dynamically tune interactions between limbic-cortical circuits through modulation of rhythmic network activity.

Sensitivity of locus ceruleus neurons to reward value for goal-directed actions

The noradrenergic nucleus locus ceruleus (LC) is associated classically with arousal and attention. Recent data suggest that it might also play a role in motivation. To study how LC neuronal responses are related to motivational intensity, we recorded 121 single neurons from two monkeys while reward size (one, two, or four drops) and the manner of obtaining reward (passive vs active) were both manipulated. The monkeys received reward under three conditions: (1) releasing a bar when a visual target changed color; (2) passively holding a bar; or (3) touching and releasing a bar. In the first two conditions, a visual cue indicated the size of the upcoming reward, and, in the third, the reward was constant through each block of 25 trials. Performance levels and lipping intensity (an appetitive behavior) both showed that the monkeys' motivation in the task was related to the predicted reward size. In conditions 1 and 2, LC neurons were activated phasically in relation to cue onset, and this activation strengthened with increasing expected reward size. In conditions 1 and 3, LC neurons were activated before the bar-release action, and the activation weakened with increasing expected reward size but only in task 1. These effects evolved as monkeys progressed through behavioral sessions, because increasing fatigue and satiety presumably progressively decreased the value of the upcoming reward. These data indicate that LC neurons integrate motivationally relevant information: both external cues and internal drives. The LC might provide the impetus to act when the predicted outcome value is low.

Face inversion decreased information about facial identity and expression in face-responsive neurons in macaque area TE

To investigate the effect of face inversion and thatcherization (eye inversion) on temporal processing stages of facial information, single neuron activities in the temporal cortex (area TE) of two rhesus monkeys were recorded. Test stimuli were colored pictures of monkey faces (four with four different expressions), human faces (three with four different expressions), and geometric shapes. Modifications were made in each face-picture, and its four variations were used as stimuli: upright original, inverted original, upright thatcherized, and inverted thatcherized faces. A total of 119 neurons responded to at least one of the upright original facial stimuli. A majority of the neurons (71%) showed activity modulations depending on upright and inverted presentations, and a lesser number of neurons (13%) showed activity modulations depending on original and thatcherized face conditions. In the case of face inversion, information about the fine category (facial identity and expression) decreased, whereas information about the global category (monkey vs human vs shape) was retained for both the original and thatcherized faces. Principal component analysis on the neuronal population responses revealed that the global categorization occurred regardless of the face inversion and that the inverted faces were represented near the upright faces in the principal component analysis space. By contrast, the face inversion decreased the ability to represent human facial identity and monkey facial expression. Thus, the neuronal population represented inverted faces as faces but failed to represent the identity and expression of the inverted faces, indicating that the neuronal representation in area TE cause the perceptual effect of face inversion.

Dopamine neurons encode errors in predicting movement trigger occurrence

The capacity to anticipate the timing of events in a dynamic environment allows us to optimize the processes necessary for perceiving, attending to, and responding to them. Such anticipation requires neuronal mechanisms that track the passage of time and use this representation, combined with prior experience, to estimate the likelihood that an event will occur (i.e., the event's "hazard rate"). Although hazard-like ramps in activity have been observed in several cortical areas in preparation for movement, it remains unclear how such time-dependent probabilities are estimated to optimize response performance. We studied the spiking activity of dopamine neurons in the substantia nigra pars compacta of monkeys during an arm-reaching task for which the foreperiod preceding the "go" signal varied randomly along a uniform distribution. After extended training, the monkeys' reaction times correlated inversely with foreperiod duration, reflecting a progressive anticipation of the go signal according to its hazard rate. Many dopamine neurons modulated their firing rates as predicted by a succession of hazard-related prediction errors. First, as time passed during the foreperiod, slowly decreasing anticipatory activity tracked the elapsed time as if encoding negative prediction errors. Then, when the go signal appeared, a phasic response encoded the temporal unpredictability of the event, consistent with a positive prediction error. Neither the anticipatory nor the phasic signals were affected by the anticipated magnitudes of future reward or effort, or by parameters of the subsequent movement. These results are consistent with the notion that dopamine neurons encode hazard-related prediction errors independently of other information.

New perspectives on catecholaminergic regulation of executive circuits: evidence for independent modulation of prefrontal functions by midbrain dopaminergic and noradrenergic neurons

Cognitive functions associated with prefrontal cortex (PFC), such as working memory and attention, are strongly influenced by catecholamine [dopamine (DA) and norepinephrine (NE)] release. Midbrain dopaminergic neurons in the ventral tegmental area and noradrenergic neurons in the locus coeruleus are major sources of DA and NE to the PFC. It is traditionally believed that DA and NE neurons are homogeneous with highly divergent axons innervating multiple terminal fields and once released, DA and NE individually or complementarily modulate the prefrontal functions and other brain regions. However, recent studies indicate that both DA and NE neurons in the mammalian brain are heterogeneous with a great degree of diversity, including their developmental lineages, molecular phenotypes, projection targets, afferent inputs, synaptic connectivity, physiological properties, and behavioral functions. These diverse characteristics could potentially endow DA and NE neurons with distinct roles in executive function, and alterations in their responses to genetic and epigenetic risk factors during development may contribute to distinct phenotypic and functional changes in disease states. In this review of recent literature, we discuss how these advances in DA and NE neurons change our thinking of catecholamine influences in cognitive functions in the brain, especially functions related to PFC. We review how the projection-target specific populations of neurons in these two systems execute their functions in both normal and abnormal conditions. Additionally, we explore what open questions remain and suggest where future research needs to move in order to provide a novel insight into the cause of neuropsychiatric disorders related to DA and NE systems.

Stimulus-related activity during conditional associations in monkey perirhinal cortex neurons depends on upcoming reward outcome

Acquiring the significance of events based on reward-related information is critical for animals to survive and to conduct social activities. The importance of the perirhinal cortex for reward-related information processing has been suggested. To examine whether or not neurons in this cortex represent reward information flexibly when a visual stimulus indicates either a rewarded or unrewarded outcome, neuronal activity in the macaque perirhinal cortex was examined using a conditional-association cued-reward task. The task design allowed us to study how the neuronal responses depended on the animal's prediction of whether it would or would not be rewarded. Two visual stimuli, a color stimulus as Cue1 followed by a pattern stimulus as Cue2, were sequentially presented. Each pattern stimulus was conditionally associated with both rewarded and unrewarded outcomes depending on the preceding color stimulus. We found an activity depending upon the two reward conditions during Cue2, i.e., pattern stimulus presentation. The response appeared after the response dependent upon the image identity of Cue2. The response delineating a specific cue sequence also appeared between the responses dependent upon the identity of Cue2 and reward conditions. Thus, when Cue1 sets the context for whether or not Cue2 indicates a reward, this region represents the meaning of Cue2, i.e., the reward conditions, independent of the identity of Cue2. These results suggest that neurons in the perirhinal cortex do more than associate a single stimulus with a reward to achieve flexible representations of reward information.