Learning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity

Striatal Dopamine Contributions to Skilled Motor Learning

Coordinated multijoint limb and digit movements-"manual dexterity"-underlie both specialized skills (e.g., playing the piano) and more mundane tasks (e.g., tying shoelaces). Impairments in dexterous skill cause significant disability, as occurs with motor cortical injury, Parkinson's disease, and a range of other pathologies. Clinical observations, as well as basic investigations, suggest that corticostriatal circuits play a critical role in learning and performing dexterous skills. Furthermore, dopaminergic signaling in these regions is implicated in synaptic plasticity and motor learning. Nonetheless, the role of striatal dopamine signaling in skilled motor learning remains poorly understood. Here, we use fiber photometry paired with a genetically encoded dopamine sensor to investigate striatal dopamine release in both male and female mice as they learn and perform a skilled reaching task. Dopamine rapidly increases during a skilled reach and peaks near pellet consumption. In the dorsolateral striatum, dopamine dynamics are faster than in the dorsomedial and ventral striatum. Across training, as reaching performance improves, dopamine signaling shifts from pellet consumption to cues that predict pellet availability, particularly in medial and ventral areas of the striatum. Furthermore, performance prediction errors are present across the striatum, with reduced dopamine release after an unsuccessful reach. These findings show that dopamine dynamics during skilled motor behaviors change with learning and are differentially regulated across striatal subregions.

Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning

Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward rewards. Dopamine (DA) neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) are crucial for this process, via engagement of a reciprocally connected network with their striatal targets. Critically, it remains unknown how dopamine neuron activity itself engages dopamine signals throughout the striatum, across learning. Here, we investigated how optogenetic Pavlovian cue conditioning of VTA or SNc dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. Although VTA stimulation-evoked robust dopamine release in NAc core, shell, and DMS, predictive cues preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS. Despite the development of vigorous movement late in training, conditioned dopamine signals did not emerge in the DLS, even during Pavlovian conditioning with SNc dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show a broad dissociation in the fundamental prediction and reward-related information generated by VTA and SNc dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale adaptations across the striatal network emerge during learning to coordinate behavior.

A neural circuit architecture for rapid learning in goal-directed navigation

Anchoring goals to spatial representations enables flexible navigation but is challenging in novel environments when both representations must be acquired simultaneously. We propose a framework for how Drosophila uses internal representations of head direction (HD) to build goal representations upon selective thermal reinforcement. We show that flies use stochastically generated fixations and directed saccades to express heading preferences in an operant visual learning paradigm and that HD neurons are required to modify these preferences based on reinforcement. We used a symmetric visual setting to expose how flies' HD and goal representations co-evolve and how the reliability of these interacting representations impacts behavior. Finally, we describe how rapid learning of new goal headings may rest on a behavioral policy whose parameters are flexible but whose form is genetically encoded in circuit architecture. Such evolutionarily structured architectures, which enable rapidly adaptive behavior driven by internal representations, may be relevant across species.

Behavioral proxies compete by the time courses of their rewards, including endogenous rewards

Natural selection is slow, so behavioral goals must be based on patterns of reward. Addictions are rewarded in the same way as adaptive choice, so they can be distinguished only by their time course. In addition, the reward process is more plastic than is generally recognized, so abstract goals are shaped by the "legibility" of their proxies.

Distributional coding of associative learning in discrete populations of midbrain dopamine neurons

Midbrain dopamine neurons are thought to play key roles in learning by conveying the difference between expected and actual outcomes. Recent evidence suggests diversity in dopamine signaling, yet it remains poorly understood how heterogeneous signals might be organized to facilitate the role of downstream circuits mediating distinct aspects of behavior. Here, we investigated the organizational logic of dopaminergic signaling by recording and labeling individual midbrain dopamine neurons during associative behavior. Our findings show that reward information and behavioral parameters are not only heterogeneously encoded but also differentially distributed across populations of dopamine neurons. Retrograde tracing and fiber photometry suggest that populations of dopamine neurons projecting to different striatal regions convey distinct signals. These data, supported by computational modeling, indicate that such distributional coding can maximize dynamic range and tailor dopamine signals to facilitate specialized roles of different striatal regions.

Integrating across behaviors and timescales to understand the neural control of movement

The nervous system evolved to enable navigation throughout the environment in the pursuit of resources. Evolutionarily newer structures allowed increasingly complex adaptations but necessarily added redundancy. A dominant view of movement neuroscientists is that there is a one-to-one mapping between brain region and function. However, recent experimental data is hard to reconcile with the most conservative interpretation of this framework, suggesting a degree of functional redundancy during the performance of well-learned, constrained behaviors. This apparent redundancy likely stems from the bidirectional interactions between the various cortical and subcortical structures involved in motor control. We posit that these bidirectional connections enable flexible interactions across structures that change depending upon behavioral demands, such as during acquisition, execution or adaptation of a skill. Observing the system across both multiple actions and behavioral timescales can help isolate the functional contributions of individual structures, leading to an integrated understanding of the neural control of movement.

Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning

Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward acquisition of rewards. Dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SNC) are crucial for this process. Dopamine neurons are embedded in a reciprocally connected network with their striatal targets, the functional organization of which remains poorly understood. Here, we investigated how learning during optogenetic Pavlovian cue conditioning of VTA or SNC dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent dopamine biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. While VTA stimulation evoked robust dopamine release in NAc core, shell, and DMS, cues predictive of this activation preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Corresponding negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS, and not the shell. Despite development of vigorous movement late in training, conditioned dopamine signals did not similarly emerge in the DLS, even during Pavlovian conditioning with SNC dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show broad dissociation in the fundamental prediction and reward-related information generated by different dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale plasticity across the striatal network emerges during Pavlovian learning to coordinate behavior.

Muscarinic Receptor Activators as Novel Treatments for Schizophrenia

Achieving optimal treatment outcomes for individuals living with schizophrenia remains challenging, despite 70 years of drug development efforts. Many chemically distinct antipsychotics have been developed over the past 7 decades with improved safety and tolerability but with only slight variation in efficacy. All antipsychotics currently approved for the treatment of schizophrenia act as antagonists or partial agonists at the dopamine D2 receptor. With only a few possible exceptions, antipsychotic drugs have similar and modest efficacy for treating positive symptoms and are relatively ineffective in addressing the negative and cognitive symptoms of the disease. The development of novel treatments focused on targeting muscarinic acetylcholine receptors (mAChRs) has been of interest for more than 25 years following reports that treatment with a dual M1/M4-preferring mAChR agonist resulted in antipsychotic-like effects and procognitive properties in individuals living with Alzheimer's disease and schizophrenia; more recent clinical trials have confirmed these findings. In addition, advances in our understanding of the receptor binding and activation properties of xanomeline at specific mAChRs have the potential to inform future drug design targeting mAChRs.

Striatal dopamine contributions to skilled motor learning

Coordinated multi-joint limb and digit movements - "manual dexterity" - underlie both specialized skills (e.g., playing the piano) and more mundane tasks (e.g., tying shoelaces). Impairments in dexterous skill cause significant disability, as occurs with motor cortical injury, Parkinson's Disease, and a range of other pathologies. Clinical observations, as well as basic investigations, suggest that cortico-striatal circuits play a critical role in learning and performing dexterous skills. Furthermore, dopaminergic signaling in these regions is implicated in synaptic plasticity and motor learning. Nonetheless, the role of striatal dopamine signaling in skilled motor learning remains poorly understood. Here, we use fiber photometry paired with a genetically encoded dopamine sensor to investigate striatal dopamine release as mice learn and perform a skilled reaching task. Dopamine rapidly increases during a skilled reach and peaks near pellet consumption. In dorsolateral striatum, dopamine dynamics are faster than in dorsomedial and ventral striatum. Across training, as reaching performance improves, dopamine signaling shifts from pellet consumption to cues that predict pellet availability, particularly in medial and ventral areas of striatum. Furthermore, performance prediction errors are present across the striatum, with reduced dopamine release after an unsuccessful reach. These findings show that dopamine dynamics during skilled motor behaviors change with learning and are differentially regulated across striatal subregions.

State and rate-of-change encoding in parallel mesoaccumbal dopamine pathways

The nervous system uses fast- and slow-adapting sensory detectors in parallel to enable neuronal representations of external states and their temporal dynamics. It is unknown whether this dichotomy also applies to internal representations that have no direct correlation in the physical world. Here we find that two distinct dopamine (DA) neuron subtypes encode either a state or its rate-of-change. In mice performing a reward-seeking task, we found that the animal's behavioral state and rate-of-change were encoded by the sustained activity of DA neurons in medial ventral tegmental area (VTA) DA neurons and transient activity in lateral VTA DA neurons, respectively. The neural activity patterns of VTA DA cell bodies matched DA release patterns within anatomically defined mesoaccumbal pathways. Based on these results, we propose a model in which the DA system uses two parallel lines for proportional-differential encoding of a state variable and its temporal dynamics.

Sensory Cues Potentiate VTA Dopamine Mediated Reinforcement

Sensory cues are critical for shaping decisions and invigorating actions during reward seeking. Dopamine neurons in the ventral tegmental area (VTA) are central in this process, supporting associative learning in Pavlovian and instrumental settings. Studies of intracranial self-stimulation (ICSS) behavior, which show that animals will work hard to receive stimulation of dopamine neurons, support the notion that dopamine transmits a reward or value signal to support learning. Recent studies have begun to question this, however, emphasizing dopamine's value-free functions, leaving its contribution to behavioral reinforcement somewhat muddled. Here, we investigated the role of sensory stimuli in dopamine-mediated reinforcement, using an optogenetic ICSS paradigm in tyrosine hydroxylase (TH)-Cre rats. We find that while VTA dopamine neuron activation in the absence of explicit external cues is sufficient to maintain robust self-stimulation, the presence of cues dramatically potentiates ICSS behavior. Our results support a framework where dopamine can have some base value as a reinforcer, but the impact of this signal is modulated heavily by the sensory learning context.

Dopaminergic reinforcement in the motor system: Implications for Parkinson's disease and deep brain stimulation

Millions of people suffer from dopamine-related disorders spanning disturbances in movement, cognition and emotion. These changes are often attributed to changes in striatal dopamine function. Thus, understanding how dopamine signalling in the striatum and basal ganglia shapes human behaviour is fundamental to advancing the treatment of affected patients. Dopaminergic neurons innervate large-scale brain networks, and accordingly, many different roles for dopamine signals have been proposed, such as invigoration of movement and tracking of reward contingencies. The canonical circuit architecture of cortico-striatal loops sparks the question, of whether dopamine signals in the basal ganglia serve an overarching computational principle. Such a holistic understanding of dopamine functioning could provide new insights into symptom generation in psychiatry to neurology. Here, we review the perspective that dopamine could bidirectionally control neural population dynamics, increasing or decreasing their strength and likelihood to reoccur in the future, a process previously termed neural reinforcement. We outline how the basal ganglia pathways could drive strengthening and weakening of circuit dynamics and discuss the implication of this hypothesis on the understanding of motor signs of Parkinson's disease (PD), the most frequent dopaminergic disorder. We propose that loss of dopamine in PD may lead to a pathological brain state where repetition of neural activity leads to weakening and instability, possibly explanatory for the fact that movement in PD deteriorates with repetition. Finally, we speculate on how therapeutic interventions such as deep brain stimulation may be able to reinstate reinforcement signals and thereby improve treatment strategies for PD in the future.

Dopamine Signaling in Substantia Nigra and Its Impact on Locomotor Function-Not a New Concept, but Neglected Reality

The mechanistic influences of dopamine (DA) signaling and impact on motor function are nearly always interpreted from changes in nigrostriatal neuron terminals in striatum. This is a standard practice in studies of human Parkinson's disease (PD) and aging and related animal models of PD and aging-related parkinsonism. However, despite dozens of studies indicating an ambiguous relationship between changes in striatal DA signaling and motor phenotype, this perseverating focus on striatum continues. Although DA release in substantia nigra (SN) was first reported almost 50 years ago, assessment of nigral DA signaling changes in relation to motor function is rarely considered. Whereas DA signaling has been well-characterized in striatum at all five steps of neurotransmission (biosynthesis and turnover, storage, release, reuptake, and post-synaptic binding) in the nigrostriatal pathway, the depth of such interrogations in the SN, outside of cell counts, is sparse. However, there is sufficient evidence that these steps in DA neurotransmission in the SN are operational and regulated autonomously from striatum and are present in human PD and aging and related animal models. To complete our understanding of how nigrostriatal DA signaling affects motor function, it is past time to include interrogation of nigral DA signaling. This brief review highlights evidence that changes in nigral DA signaling at each step in DA neurotransmission are autonomous from those in striatum and changes in the SN alone can influence locomotor function. Accordingly, for full characterization of how nigrostriatal DA signaling affects locomotor activity, interrogation of DA signaling in SN is essential.

Adolescent alcohol drinking interaction with the gut microbiome: implications for adult alcohol use disorder

Reciprocal communication between the gut microbiota and the brain, commonly referred to as the "gut-brain-axis" is crucial in maintaining overall physiological homeostasis. Gut microbiota development and brain maturation (neuronal connectivity and plasticity) appear to be synchronized and to follow the same timeline during childhood (immature), adolescence (expansion) and adulthood (completion). It is important to note that the mesolimbic reward circuitry develops early on, whereas the maturation of the inhibitory frontal cortical neurons is delayed. This imbalance can lead to increased acquirement of reward-seeking and risk-taking behaviors during adolescence, and consequently eventuate in heightened risk for substance abuse. Thus, there is high initiation of alcohol drinking in early adolescence that significantly increases the risk of alcohol use disorder (AUD) in adulthood. The underlying causes for heightened AUD risk are not well understood. It is suggested that alcohol-associated gut microbiota impairment during adolescence plays a key role in AUD neurodevelopment in adulthood. Furthermore, alcohol-induced dysregulation of microglia, either directly or indirectly through interaction with gut microbiota, may be a critical neuroinflammatory pathway leading to neurodevelopmental impairments and AUD. In this review article, we highlight the influence of adolescent alcohol drinking on gut microbiota, gut-brain axis and microglia, and eventual manifestation of AUD. Furthermore, novel therapeutic interventions via gut microbiota manipulations are discussed briefly.

Atomoxetine and reward size equally improve task engagement and perceptual decisions but differently affect movement execution

Our ability to engage and perform daily activities relies on balancing the associated benefits and costs. Rewards, as benefits, act as powerful motivators that help us stay focused for longer durations. The noradrenergic (NA) system is thought to play a significant role in optimizing our performance. Yet, the interplay between reward and the NA system in shaping performance remains unclear, particularly when actions are driven by external incentives (reward). To explore this interaction, we tested four female rhesus monkeys performing a sustained Go/NoGo task under two reward sizes (low/high) and three pharmacological conditions (saline and two doses of atomoxetine, a NA reuptake inhibitor: ATX-0.5 mg/kg and ATX-1 mg/kg). We found that increasing either reward or NA levels equally enhanced the animal's engagement in the task compared to low reward saline; the animals also responded faster and more consistently under these circumstances. Notably, we identified differences between reward size and ATX. When combined with ATX, high reward further reduced the occurrence of false alarms (i.e., incorrect go trials on distractors), implying that it helped further suppress impulsive responses. In addition, ATX (but not reward size) consistently increased movement duration dose-dependently, while high reward did not affect movement duration but decreased its variability. We conclude that noradrenaline and reward modulate performance, but their effects are not identical, suggesting differential underlying mechanisms. Reward might energize/invigorate decisions and action, while ATX might help regulate energy expenditure, depending on the context, through the NA system.

Stress relief as a natural resilience mechanism against depression-like behaviors

Relief, the appetitive state after the termination of aversive stimuli, is evolutionarily conserved. Understanding the behavioral role of this well-conserved phenomenon and its underlying neurobiological mechanisms are open and important questions. Here, we discover that the magnitude of relief from physical stress strongly correlates with individual resilience to depression-like behaviors in chronic stressed mice. Notably, blocking stress relief causes vulnerability to depression-like behaviors, whereas natural rewards supplied shortly after stress promotes resilience. Stress relief is mediated by reward-related mesolimbic dopamine neurons, which show minute-long, persistent activation after stress termination. Circuitry-wise, activation or inhibition of circuits downstream of the ventral tegmental area during the transient relief period bi-directionally regulates depression resilience. These results reveal an evolutionary function of stress relief in depression resilience and identify the neural substrate mediating this effect. Importantly, our data suggest a behavioral strategy of augmenting positive valence of stress relief with natural rewards to prevent depression.

Distributed dopaminergic signaling in the basal ganglia and its relationship to motor disability in Parkinson's disease

The degeneration of mesencephalic dopaminergic neurons that innervate the basal ganglia is responsible for the cardinal motor symptoms of Parkinson's disease (PD). It has been thought that loss of dopaminergic signaling in one basal ganglia region - the striatum - was solely responsible for the network pathophysiology causing PD motor symptoms. While our understanding of dopamine (DA)'s role in modulating striatal circuitry has deepened in recent years, it also has become clear that it acts in other regions of the basal ganglia to influence movement. Underscoring this point, examination of a new progressive mouse model of PD shows that striatal dopamine DA depletion alone is not sufficient to induce parkinsonism and that restoration of extra-striatal DA signaling attenuates parkinsonian motor deficits once they appear. This review summarizes recent advances in the effort to understand basal ganglia circuitry, its modulation by DA, and how its dysfunction drives PD motor symptoms.

Nigrostriatal dopamine modulates the striatal-amygdala pathway in auditory fear conditioning

The auditory striatum, a sensory portion of the dorsal striatum, plays an essential role in learning and memory. In contrast to its roles and underlying mechanisms in operant conditioning, however, little is known about its contribution to classical auditory fear conditioning. Here, we reveal the function of the auditory striatum in auditory-conditioned fear memory. We find that optogenetically inhibiting auditory striatal neurons impairs fear memory formation, which is mediated through the striatal-amygdala pathway. Using calcium imaging in behaving mice, we find that auditory striatal neuronal responses to conditioned tones potentiate across memory acquisition and expression. Furthermore, nigrostriatal dopaminergic projections plays an important role in modulating conditioning-induced striatal potentiation. Together, these findings demonstrate the existence of a nigro-striatal-amygdala circuit for conditioned fear memory formation and expression.

Reward prediction-errors weighted by cue salience produces addictive behaviours in simulations, with asymmetrical learning and steeper delay discounting

Dysfunction in learning and motivational systems are thought to contribute to addictive behaviours. Previous models have suggested that dopaminergic roles in learning and motivation could produce addictive behaviours through pharmacological manipulations that provide excess dopaminergic signalling towards these learning and motivational systems. Redish (2004) suggested a role based on dopaminergic signals of value prediction error, while (Zhang et al., 2009) suggested a role based on dopaminergic signals of motivation. However, both models present significant limitations. They do not explain the reduced sensitivity to drug-related costs/negative consequences, the increased impulsivity generally found in people with a substance use disorder, craving behaviours, and non-pharmacological dependence, all of which are key hallmarks of addictive behaviours. Here, we propose a novel mathematical definition of salience, that combines aspects of dopamine's role in both learning and motivation within the reinforcement learning framework. Using a single parameter regime, we simulated addictive behaviours that the (Zhang et al., 2009; Redish, 2004) models also produce but we went further in simulating the downweighting of drug-related negative prediction-errors, steeper delay discounting of drug rewards, craving behaviours and aspects of behavioural/non-pharmacological addictions. The current salience model builds on our recently proposed conceptual theory that salience modulates internal representation updating and may contribute to addictive behaviours by producing misaligned internal representations (Kalhan et al., 2021). Critically, our current mathematical model of salience argues that the seemingly disparate learning and motivational aspects of dopaminergic functioning may interact through a salience mechanism that modulates internal representation updating.

Sensory cues potentiate VTA dopamine mediated reinforcement

Sensory cues are critical for shaping decisions and invigorating actions during reward seeking. Dopamine neurons in the ventral tegmental area (VTA) are critical in this process, supporting associative learning in Pavlovian and instrumental settings. Studies of intracranial self stimulation (ICSS) behavior, which show that animals will work hard to receive stimulation of dopamine neurons, support the notion that dopamine transmits a reward or value signal to support learning. Recent studies have begun to question this, however, emphasizing dopamine's value-free functions, leaving its contribution to behavioral reinforcement somewhat muddled. Here, we investigated the role of sensory stimuli in dopamine-mediated reinforcement, using an optogenetic ICSS paradigm in tyrosine hydroxylase (TH)-cre rats. We find that while VTA dopamine neuron activation in the absence of any external cueing stimulus is sufficient to maintain robust self stimulation, the presence of cues dramatically potentiates ICSS behavior. Our results support a framework where dopamine can have some base value as a reinforcer, but the impact of this signal is modulated heavily by the sensory learning context.

Unique functional responses differentially map onto genetic subtypes of dopamine neurons

Dopamine neurons are characterized by their response to unexpected rewards, but they also fire during movement and aversive stimuli. Dopamine neuron diversity has been observed based on molecular expression profiles; however, whether different functions map onto such genetic subtypes remains unclear. In this study, we established that three genetic dopamine neuron subtypes within the substantia nigra pars compacta, characterized by the expression of Slc17a6 (Vglut2), Calb1 and Anxa1, each have a unique set of responses to rewards, aversive stimuli and accelerations and decelerations, and these signaling patterns are highly correlated between somas and axons within subtypes. Remarkably, reward responses were almost entirely absent in the Anxa1+ subtype, which instead displayed acceleration-correlated signaling. Our findings establish a connection between functional and genetic dopamine neuron subtypes and demonstrate that molecular expression patterns can serve as a common framework to dissect dopaminergic functions.

Recording neural reward signals in the real-world using mobile-EEG and augmented reality

The electrophysiological response to rewards recorded during laboratory-based tasks has been well documented over the past two decades, yet little is known about the neural response patterns in 'real-world' settings. To address this issue, we combined a mobile-EEG system with an augmented reality headset (which blends high definition "holograms" within the real-world) to record event-related brain potentials (ERP) while participants navigated an operant chamber to find rewards. 25 participants (age = 18-43, Male=6, Female=19) were asked to choose between two floating holograms marking a west or east goal-location in a large room, and once participants reached the goal location, the hologram would turn into a reward (5 cents) or no-reward (0 cents) cue. Following the feedback cue, participants were required to return to a hologram marking the start location, and once standing in it, a 3 second counter hologram would initiate the next trial. This sequence was repeated until participants completed 200 trials. Consistent with previous research, reward feedback evoked the reward positivity, an ERP component believed to index the sensitivity of the anterior cingulate cortex to reward prediction error signals. The reward positivity peaked around 235ms post-feedback with a maximal at channel FCz (M=-2.60μV, SD=1.73μV) and was significantly different than zero (p < 0.01). At a behavioral level, participants took approximately 3.38 seconds to reach the goal-location and exhibited a general lose-shift (68.3% ± 3.5) response strategy and were slightly slower to return to the start location following negative feedback (2.43 sec) compared to positive feedback (2.38 sec), evidence of post-error slowing. Overall, these findings provide the first evidence that combining mobile-EEG with augmented reality technology is a feasible solution to enhance the ecological validity of human electrophysiological studies of goal-directed behavior and a step towards a new era of human cognitive neuroscience research that blurs the line between laboratory and reality.

Intrinsic dopamine and acetylcholine dynamics in the striatum of mice

External rewards such as food and money are potent modifiers of behaviour1,2. Pioneering studies established that these salient sensory stimuli briefly interrupt the tonic discharge of neurons that produce the neuromodulators dopamine (DA) and acetylcholine (ACh): midbrain DA neurons (DANs) fire a burst of action potentials that broadly elevates DA in the striatum3,4 at the same time that striatal cholinergic interneurons (CINs) produce a characteristic pause in firing5,6. These phasic responses are thought to create unique, temporally limited conditions that motivate action and promote learning7-11. However, the dynamics of DA and ACh outside explicitly rewarded situations remain poorly understood. Here we show that extracellular DA and ACh levels fluctuate spontaneously and periodically at a frequency of approximately 2 Hz in the dorsal striatum of mice and maintain the same temporal relationship relative to one another as that evoked by reward. We show that this neuromodulatory coordination does not arise from direct interactions between DA and ACh within the striatum. Instead, we provide evidence that periodic fluctuations in striatal DA are inherited from midbrain DANs, while striatal ACh transients are driven by glutamatergic inputs, which act to locally synchronize the spiking of CINs. Together, our findings show that striatal neuromodulatory dynamics are autonomously organized by distributed extra-striatal afferents. The dominance of intrinsic rhythms in DA and ACh offers new insights for explaining how reward-associated neural dynamics emerge and how the brain motivates action and promotes learning from within.

Rethinking the network determinants of motor disability in Parkinson's disease

For roughly the last 30 years, the notion that striatal dopamine (DA) depletion was the critical determinant of network pathophysiology underlying the motor symptoms of Parkinson's disease (PD) has dominated the field. While the basal ganglia circuit model underpinning this hypothesis has been of great heuristic value, the hypothesis itself has never been directly tested. Moreover, studies in the last couple of decades have made it clear that the network model underlying this hypothesis fails to incorporate key features of the basal ganglia, including the fact that DA acts throughout the basal ganglia, not just in the striatum. Underscoring this point, recent work using a progressive mouse model of PD has shown that striatal DA depletion alone is not sufficient to induce parkinsonism and that restoration of extra-striatal DA signaling attenuates parkinsonian motor deficits once they appear. Given the broad array of discoveries in the field, it is time for a new model of the network determinants of motor disability in PD.

Dead rats, dopamine, performance metrics, and peacock tails: Proxy failure is an inherent risk in goal-oriented systems

When a measure becomes a target, it ceases to be a good measure. For example, when standardized test scores in education become targets, teachers may start "teaching to the test," leading to breakdown of the relationship between the measure - test performance - and the underlying goal - quality education. Similar phenomena have been named and described across a broad range of contexts, such as economics, academia, machine learning, and ecology. Yet it remains unclear whether these phenomena bear only superficial similarities, or if they derive from some fundamental unifying mechanism. Here, we propose such a unifying mechanism, which we label proxy failure. We first review illustrative examples and their labels, such as the "cobra effect," "Goodhart's law," and "Campbell's law." Second, we identify central prerequisites and constraints of proxy failure, noting that it is often only a partial failure or divergence. We argue that whenever incentivization or selection is based on an imperfect proxy measure of the underlying goal, a pressure arises that tends to make the proxy a worse approximation of the goal. Third, we develop this perspective for three concrete contexts, namely neuroscience, economics, and ecology, highlighting similarities and differences. Fourth, we outline consequences of proxy failure, suggesting it is key to understanding the structure and evolution of goal-oriented systems. Our account draws on a broad range of disciplines, but we can only scratch the surface within each. We thus hope the present account elicits a collaborative enterprise, entailing both critical discussion as well as extensions in contexts we have missed.

Dynamic modulation of subthalamic nucleus activity facilitates adaptive behavior

Adapting actions to changing goals and environments is central to intelligent behavior. There is evidence that the basal ganglia play a crucial role in reinforcing or adapting actions depending on their outcome. However, the corresponding electrophysiological correlates in the basal ganglia and the extent to which these causally contribute to action adaptation in humans is unclear. Here, we recorded electrophysiological activity and applied bursts of electrical stimulation to the subthalamic nucleus, a core area of the basal ganglia, in 16 patients with Parkinson's disease (PD) on medication using temporarily externalized deep brain stimulation (DBS) electrodes. Patients as well as 16 age- and gender-matched healthy participants attempted to produce forces as close as possible to a target force to collect a maximum number of points. The target force changed over trials without being explicitly shown on the screen so that participants had to infer target force based on the feedback they received after each movement. Patients and healthy participants were able to adapt their force according to the feedback they received (P < 0.001). At the neural level, decreases in subthalamic beta (13 to 30 Hz) activity reflected poorer outcomes and stronger action adaptation in 2 distinct time windows (Pcluster-corrected < 0.05). Stimulation of the subthalamic nucleus reduced beta activity and led to stronger action adaptation if applied within the time windows when subthalamic activity reflected action outcomes and adaptation (Pcluster-corrected < 0.05). The more the stimulation volume was connected to motor cortex, the stronger was this behavioral effect (Pcorrected = 0.037). These results suggest that dynamic modulation of the subthalamic nucleus and interconnected cortical areas facilitates adaptive behavior.

The Supplementary Motor Area and Automatic Cognitive Control: Lack of Evidence from Two Neuromodulation Techniques

The SMA is fundamental in planning voluntary movements and execution of some cognitive control operations. Specifically, the SMA has been known to play a dominant role in controlling goal-directed actions as well as those that are highly predicted (i.e., automatic). Yet, the essential contribution of SMA in goal-directed or automatic control of behavior is scarce. Our objective was to test the possible direct role of SMA in automatic and voluntary response inhibition. We separately applied two noninvasive brain stimulation (NIBS) inhibitory techniques over SMA: either continuous theta-burst stimulation using repetitive transcranial magnetic stimulation or transcranial static magnetic field stimulation. Each NIBS technique was performed in a randomized, crossover, sham-controlled design. Before applying NIBS, participants practiced a go/no-go learning task where associations between stimulus and stopping behaviors were created (initiation and inhibition). After applying each NIBS, participants performed a go/no-go task with reversed associations (automatic control) and the stop signal task (voluntary control). Learning associations between stimuli and response initiation/inhibition was achieved by participants and therefore automatized during training. However, no significant differences between real and sham NIBS were found in either automatic (go/no-go learning task) or voluntary inhibition (stop signal task), with Bayesian statistics providing moderate evidence of absence. In conclusion, our results are compatible with a nondirect involvement of SMA in automatic control of behavior. Further studies are needed to prove a noncausal link between prior neuroimaging findings relative to SMA controlling functions and the observed behavior.

Mesolimbic dopamine adapts the rate of learning from action

Recent success in training artificial agents and robots derives from a combination of direct learning of behavioural policies and indirect learning through value functions^1-3. Policy learning and value learning use distinct algorithms that optimize behavioural performance and reward prediction, respectively. In animals, behavioural learning and the role of mesolimbic dopamine signalling have been extensively evaluated with respect to reward prediction⁴; however, so far there has been little consideration of how direct policy learning might inform our understanding⁵. Here we used a comprehensive dataset of orofacial and body movements to understand how behavioural policies evolved as naive, head-restrained mice learned a trace conditioning paradigm. Individual differences in initial dopaminergic reward responses correlated with the emergence of learned behavioural policy, but not the emergence of putative value encoding for a predictive cue. Likewise, physiologically calibrated manipulations of mesolimbic dopamine produced several effects inconsistent with value learning but predicted by a neural-network-based model that used dopamine signals to set an adaptive rate, not an error signal, for behavioural policy learning. This work provides strong evidence that phasic dopamine activity can regulate direct learning of behavioural policies, expanding the explanatory power of reinforcement learning models for animal learning⁶.

Tripartite extended amygdala-basal ganglia CRH circuit drives locomotor activation and avoidance behavior

An adaptive stress response involves various mediators and circuits orchestrating a complex interplay of physiological, emotional, and behavioral adjustments. We identified a population of corticotropin-releasing hormone (CRH) neurons in the lateral part of the interstitial nucleus of the anterior commissure (IPACL), a subdivision of the extended amygdala, which exclusively innervate the substantia nigra (SN). Specific stimulation of this circuit elicits hyperactivation of the hypothalamic-pituitary-adrenal axis, locomotor activation, and avoidance behavior contingent on CRH receptor type 1 (CRHR1) located at axon terminals in the SN, which originate from external globus pallidus (GPe) neurons. The neuronal activity prompting the observed behavior is shaped by IPACL^CRH and GPe^CRHR1 neurons coalescing in the SN. These results delineate a previously unidentified tripartite CRH circuit functionally connecting extended amygdala and basal ganglia nuclei to drive locomotor activation and avoidance behavior.

Nucleus accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error

There is active debate on the role of dopamine in processing aversive stimuli, where inferred roles range from no involvement at all, to signaling an aversive prediction error (APE). Here, we systematically investigate dopamine release in the nucleus accumbens core (NAC), which is closely linked to reward prediction errors, in rats exposed to white noise (WN, a versatile, underutilized, aversive stimulus) and its predictive cues. Both induced a negative dopamine ramp, followed by slow signal recovery upon stimulus cessation. In contrast to reward conditioning, this dopamine signal was unaffected by WN value, context valence, or probabilistic contingencies, and the WN dopamine response shifted only partially toward its predictive cue. However, unpredicted WN provoked slower post-stimulus signal recovery than predicted WN. Despite differing signal qualities, dopamine responses to simultaneous presentation of rewarding and aversive stimuli were additive. Together, our findings demonstrate that instead of an APE, NAC dopamine primarily tracks prediction and duration of aversive events.

Motor constellation theory: A model of infants' phonological development

Every normally developing human infant solves the difficult problem of mapping their native-language phonology, but the neural mechanisms underpinning this behavior remain poorly understood. Here, motor constellation theory, an integrative neurophonological model, is presented, with the goal of explicating this issue. It is assumed that infants' motor-auditory phonological mapping takes place through infants' orosensory "reaching" for phonological elements observed in the language-specific ambient phonology, via reference to kinesthetic feedback from motor systems (e.g., articulators), and auditory feedback from resulting speech and speech-like sounds. Attempts are regulated by basal ganglion-cerebellar speech neural circuitry, and successful attempts at reproduction are enforced through dopaminergic signaling. Early in life, the pace of anatomical development constrains mapping such that complete language-specific phonological mapping is prohibited by infants' undeveloped supralaryngeal vocal tract and undescended larynx; constraints gradually dissolve with age, enabling adult phonology. Where appropriate, reference is made to findings from animal and clinical models. Some implications for future modeling and simulation efforts, as well as clinical settings, are also discussed.

Nigrostriatal dopamine pathway regulates auditory discrimination behavior

The auditory striatum, the tail portion of dorsal striatum in basal ganglia, is implicated in perceptual decision-making, transforming auditory stimuli to action outcomes. Despite its known connections to diverse neurological conditions, the dopaminergic modulation of sensory striatal neuronal activity and its behavioral influences remain unknown. We demonstrated that the optogenetic inhibition of dopaminergic projections from the substantia nigra pars compacta to the auditory striatum specifically impairs mouse choice performance but not movement in an auditory frequency discrimination task. In vivo dopamine and calcium imaging in freely behaving mice revealed that this dopaminergic projection modulates striatal tone representations, and tone-evoked striatal dopamine release inversely correlated with the evidence strength of tones. Optogenetic inhibition of D1-receptor expressing neurons and pharmacological inhibition of D1 receptors in the auditory striatum dampened choice performance accuracy. Our study uncovers a phasic mechanism within the nigrostriatal system that regulates auditory decisions by modulating ongoing auditory perception.

The auditory striatum, the tail portion of dorsal striatum, is implicated in decision-making. This study uncovers a phasic mechanism within the nigrostriatal system that regulates auditory decisions by modulating ongoing auditory perception.

Mesoaccumbal Dopamine Heterogeneity: What Do Dopamine Firing and Release Have to Do with It?

Ventral tegmental area (VTA) dopamine (DA) neurons are often thought to uniformly encode reward prediction errors. Conversely, DA release in the nucleus accumbens (NAc), the prominent projection target of these neurons, has been implicated in reinforcement learning, motivation, aversion, and incentive salience. This contrast between heterogeneous functions of DA release versus a homogeneous role for DA neuron activity raises numerous questions regarding how VTA DA activity translates into NAc DA release. Further complicating this issue is increasing evidence that distinct VTA DA projections into defined NAc subregions mediate diverse behavioral functions. Here, we evaluate evidence for heterogeneity within the mesoaccumbal DA system and argue that frameworks of DA function must incorporate the precise topographic organization of VTA DA neurons to clarify their contribution to health and disease.

Serotonin receptors contribute to dopamine depression of lateral inhibition in the nucleus accumbens

Dopamine modulation of nucleus accumbens (NAc) circuitry is central to theories of reward seeking and reinforcement learning. Despite decades of effort, the acute dopamine actions on the NAc microcircuitry remain puzzling. Here, we dissect out the direct actions of dopamine on lateral inhibition between medium spiny neurons (MSNs) in mouse brain slices and find that they are pathway specific. Dopamine potently depresses GABAergic transmission from presynaptic dopamine D2 receptor-expressing MSNs (D2-MSNs), whereas it potentiates transmission from presynaptic dopamine D1 receptor-expressing MSNs (D1-MSNs) onto other D1-MSNs. To our surprise, presynaptic D2 receptors mediate only half of the depression induced by endogenous and exogenous dopamine. Presynaptic serotonin 5-HT1B receptors are responsible for a significant component of dopamine-induced synaptic depression. This study clarifies the mechanistic understanding of dopamine actions in the NAc by showing pathway-specific modulation of lateral inhibition and involvement of D2 and 5-HT1B receptors in dopamine depression of D2-MSN synapses.

Burke and Alvarez find that, in the nucleus accumbens, dopamine depresses or potentiates lateral inhibition between projection neurons, depending on the specific synapses isolated. Dopamine depression of D2-MSN GABA transmission involves activation of 5-HT1B receptors, suggesting that cross-talk between monoamines and receptors plays a role in accumbens circuit function.

The dopamine transporter gene SLC6A3: multidisease risks

The human dopamine transporter gene SLC6A3 has been consistently implicated in several neuropsychiatric diseases but the disease mechanism remains elusive. In this risk synthesis, we have concluded that SLC6A3 represents an increasingly recognized risk with a growing number of familial mutants associated with neuropsychiatric and neurological disorders. At least five loci were related to common and severe diseases including alcohol use disorder (high activity variant), attention-deficit/hyperactivity disorder (low activity variant), autism (familial proteins with mutated networking) and movement disorders (both regulatory variants and familial mutations). Association signals depended on genetic markers used as well as ethnicity examined. Strong haplotype selection and gene-wide epistases support multimarker assessment of functional variations and phenotype associations. Inclusion of its promoter region's functional markers such as DNPi (rs67175440) and 5'VNTR (rs70957367) may help delineate condensate-based risk action, testing a locus-pathway-phenotype hypothesis for one gene-multidisease etiology.

Slowly evolving dopaminergic activity modulates the moment-to-moment probability of reward-related self-timed movements

Clues from human movement disorders have long suggested that the neurotransmitter dopamine plays a role in motor control, but how the endogenous dopaminergic system influences movement is unknown. Here we examined the relationship between dopaminergic signaling and the timing of reward-related movements in mice. Animals were trained to initiate licking after a self-timed interval following a start-timing cue; reward was delivered in response to movements initiated after a criterion time. The movement time was variable from trial-to-trial, as expected from previous studies. Surprisingly, dopaminergic signals ramped-up over seconds between the start-timing cue and the self-timed movement, with variable dynamics that predicted the movement/reward time on single trials. Steeply rising signals preceded early lick-initiation, whereas slowly rising signals preceded later initiation. Higher baseline signals also predicted earlier self-timed movements. Optogenetic activation of dopamine neurons during self-timing did not trigger immediate movements, but rather caused systematic early-shifting of movement initiation, whereas inhibition caused late-shifting, as if modulating the probability of movement. Consistent with this view, the dynamics of the endogenous dopaminergic signals quantitatively predicted the moment-by-moment probability of movement initiation on single trials. We propose that ramping dopaminergic signals, likely encoding dynamic reward expectation, can modulate the decision of when to move.

The Dopamine System and Automatization of Movement Sequences: A Review With Relevance for Speech and Stuttering

The last decades of research have gradually elucidated the complex functions of the dopamine system in the vertebrate brain. The multiple roles of dopamine in motor function, learning, attention, motivation, and the emotions have been difficult to reconcile. A broad and detailed understanding of the physiology of cerebral dopamine is of importance in understanding a range of human disorders. One of the core functions of dopamine involves the basal ganglia and the learning and execution of automatized sequences of movements. Speech is one of the most complex and highly automatized sequential motor behaviors, though the exact roles that the basal ganglia and dopamine play in speech have been difficult to determine. Stuttering is a speech disorder that has been hypothesized to be related to the functions of the basal ganglia and dopamine. The aim of this review was to provide an overview of the current understanding of the cerebral dopamine system, in particular the mechanisms related to motor learning and the execution of movement sequences. The primary aim was not to review research on speech and stuttering, but to provide a platform of neurophysiological mechanisms, which may be utilized for further research and theoretical development on speech, speech disorders, and other behavioral disorders. Stuttering and speech are discussed here only briefly. The review indicates that a primary mechanism for the automatization of movement sequences is the merging of isolated movements into chunks that can be executed as units. In turn, chunks can be utilized hierarchically, as building blocks of longer chunks. It is likely that these mechanisms apply also to speech, so that frequent syllables and words are produced as motor chunks. It is further indicated that the main learning principle for sequence learning is reinforcement learning, with the phasic release of dopamine as the primary teaching signal indicating successful sequences. It is proposed that the dynamics of the dopamine system constitute the main neural basis underlying the situational variability of stuttering.

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.

Striatal BOLD and Midfrontal Theta Power Express Motivation for Action

Action selection is biased by the valence of anticipated outcomes. To assess mechanisms by which these motivational biases are expressed and controlled, we measured simultaneous EEG-fMRI during a motivational Go/NoGo learning task (N = 36), leveraging the temporal resolution of EEG and subcortical access of fMRI. VmPFC BOLD encoded cue valence, importantly predicting trial-by-trial valence-driven response speed differences and EEG theta power around cue onset. In contrast, striatal BOLD encoded selection of active Go responses and correlated with theta power around response time. Within trials, theta power ramped in the fashion of an evidence accumulation signal for the value of making a "Go" response, capturing the faster responding to reward cues. Our findings reveal a dual nature of midfrontal theta power, with early components reflecting the vmPFC contribution to motivational biases, and late components reflecting their striatal translation into behavior, in line with influential recent "value of work" theories of striatal processing.

Neural correlates of vocal initiation in the VTA/SNc of juvenile male zebra finches

Initiation and execution of complex learned vocalizations such as human speech and birdsong depend on multiple brain circuits. In songbirds, neurons in the motor cortices and basal ganglia circuitry exhibit preparatory activity before initiation of song, and that activity is thought to play an important role in successful song performance. However, it remains unknown where a start signal for song is represented in the brain and how such a signal would lead to appropriate vocal initiation. To test whether neurons in the midbrain ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) show activity related to song initiation, we carried out extracellular recordings of VTA/SNc single units in singing juvenile male zebra finches. We found that a subset of VTA/SNc units exhibit phasic activity precisely time-locked to the onset of the song bout, and that the activity occurred specifically at the beginning of song. These findings suggest that phasic activity in the VTA/SNc represents a start signal that triggers song vocalization.

Context-dependent representations of movement in Drosophila dopaminergic reinforcement pathways

Dopamine plays a central role in motivating and modifying behavior, serving to invigorate current behavioral performance and guide future actions through learning. Here we examine how this single neuromodulator can contribute to such diverse forms of behavioral modulation. By recording from the dopaminergic reinforcement pathways of the Drosophila mushroom body during active odor navigation, we reveal how their ongoing motor-associated activity relates to goal-directed behavior. We found that dopaminergic neurons correlate with different behavioral variables depending on the specific navigational strategy of an animal, such that the activity of these neurons preferentially reflects the actions most relevant to odor pursuit. Furthermore, we show that these motor correlates are translated to ongoing dopamine release, and acutely perturbing dopaminergic signaling alters the strength of odor tracking. Context-dependent representations of movement and reinforcement cues are thus multiplexed within the mushroom body dopaminergic pathways, enabling them to coordinately influence both ongoing and future behavior.

Overexpression of miR-133b protects against isoflurane-induced learning and memory impairment

A number of microRNAs (miRs) have been identified as being involved in the regulation of anesthesia-induced cognitive impairment. The aim of the present study was to investigated the role and potential mechanism of miR-133b in isoflurane-induced learning and memory impairment. An animal model of isoflurane exposure was established using neonatal Sprague-Dawley rats. The rats were trained for Morris water maze (MWM) testing to assess their spatial learning and memory ability. Reverse transcription-quantitative polymerase chain reaction was used for the measurement of miR-133b expression in hippocampal tissues and primary hippocampal neuron cultures. Cell viability was assessed using a Cell Counting Kit-8 assay, and flow cytometric analysis was used to determine the rate of apoptosis. The MWM test results indicated that during the training period, the time required to locate the platform was significantly increased for rats exposed to isoflurane, and this increased time was reduced by the overexpression of miR-133b. The results of a probe trial indicated that isoflurane exposure increased escape latency and decreased the time spent in the platform area for isoflurane-treated rats; however, these effects were reversed by the injection of miR-133b agomir. The in vitro experiments demonstrated that the overexpression of miR-133b attenuated the reduction of neuronal cell viability induced by isoflurane, and inhibited the isoflurane-induced apoptosis of hippocampal neurons. In conclusion, the present study revealed that the overexpression of miR-133b attenuated isoflurane-induced learning and memory impairment in rats. Furthermore, miR-133b overexpression promoted the viability of hippocampal neurons and their resistance to apoptosis when exposed to isoflurane.

Mesocorticolimbic Dopamine Pathways Across Adolescence: Diversity in Development

Mesocorticolimbic dopamine circuity undergoes a protracted maturation during adolescent life. Stable adult levels of behavioral functioning in reward, motivational, and cognitive domains are established as these pathways are refined, however, their extended developmental window also leaves them vulnerable to perturbation by environmental factors. In this review, we highlight recent advances in understanding the mechanisms underlying dopamine pathway development in the adolescent brain, and how the environment influences these processes to establish or disrupt neurocircuit diversity. We further integrate these recent studies into the larger historical framework of anatomical and neurochemical changes occurring during adolescence in the mesocorticolimbic dopamine system. While dopamine neuron heterogeneity is increasingly appreciated at molecular, physiological, and anatomical levels, we suggest that a developmental facet may play a key role in establishing vulnerability or resilience to environmental stimuli and experience in distinct dopamine circuits, shifting the balance between healthy brain development and susceptibility to psychiatric disease.

Dissociable contributions of phasic dopamine activity to reward and prediction

Sensory cues that precede reward acquire predictive (expected value) and incentive (drive reward-seeking action) properties. Mesolimbic dopamine neurons' responses to sensory cues correlate with both expected value and reward-seeking action. This has led to the proposal that phasic dopamine responses may be sufficient to inform value-based decisions, elicit actions, and/or induce motivational states; however, causal tests are incomplete. Here, we show that direct dopamine neuron stimulation, both calibrated to physiological and greater intensities, at the time of reward can be sufficient to induce and maintain reward seeking (reinforcing) although replacement of a cue with stimulation is insufficient to induce reward seeking or act as an informative cue. Stimulation of descending cortical inputs, one synapse upstream, are sufficient for reinforcement and cues to future reward. Thus, physiological activation of mesolimbic dopamine neurons can be sufficient for reinforcing properties of reward without being sufficient for the predictive and incentive properties of cues.

Reinforcement learning links spontaneous cortical dopamine impulses to reward

In their pioneering study on dopamine release, Romo and Schultz speculated "...that the amount of dopamine released by unmodulated spontaneous impulse activity exerts a tonic, permissive influence on neuronal processes more actively engaged in preparation of self-initiated movements...."¹ Motivated by the suggestion of "spontaneous impulses," as well as by the "ramp up" of dopaminergic neuronal activity that occurs when rodents navigate to a reward,^2-5 we asked two questions. First, are there spontaneous impulses of dopamine that are released in cortex? Using cell-based optical sensors of extrasynaptic dopamine, [DA]_ex,⁶ we found that spontaneous dopamine impulses in cortex of naive mice occur at a rate of ∼0.01 per second. Next, can mice be trained to change the amplitude and/or timing of dopamine events triggered by internal brain dynamics, much as they can change the amplitude and timing of dopamine impulses based on an external cue?^7-9 Using a reinforcement learning paradigm based solely on rewards that were gated by feedback from real-time measurements of [DA]_ex, we found that mice can volitionally modulate their spontaneous [DA]_ex. In particular, by only the second session of daily, hour-long training, mice increased the rate of impulses of [DA]_ex, increased the amplitude of the impulses, and increased their tonic level of [DA]_ex for a reward. Critically, mice learned to reliably elicit [DA]_ex impulses prior to receiving a reward. These effects reversed when the reward was removed. We posit that spontaneous dopamine impulses may serve as a salient cognitive event in behavioral planning.

A mosaic of cost-benefit control over cortico-striatal circuitry

Dopamine contributes to cognitive control through well-established effects in both the striatum and cortex. Although earlier work suggests that dopamine affects cognitive control capacity, more recent work suggests that striatal dopamine may also impact on cognitive motivation. We consider the emerging perspective that striatal dopamine boosts control by making people more sensitive to the benefits versus the costs of cognitive effort, and we discuss how this sensitivity shapes competition between controlled and prepotent actions. We propose that dopamine signaling in distinct cortico-striatal subregions mediates different types of cost-benefit tradeoffs, and also discuss mechanisms for the local control of dopamine release, enabling selectivity among cortico-striatal circuits. In so doing, we show how this cost-benefit mosaic can reconcile seemingly conflicting findings about the impact of dopamine signaling on cognitive control.

Spine dynamics in the brain, mental disorders and artificial neural networks

In the brain, most synapses are formed on minute protrusions known as dendritic spines. Unlike their artificial intelligence counterparts, spines are not merely tuneable memory elements: they also embody algorithms that implement the brain's ability to learn from experience and cope with new challenges. Importantly, they exhibit structural dynamics that depend on activity, excitatory input and inhibitory input (synaptic plasticity or 'extrinsic' dynamics) and dynamics independent of activity ('intrinsic' dynamics), both of which are subject to neuromodulatory influences and reinforcers such as dopamine. Here we succinctly review extrinsic and intrinsic dynamics, compare these with parallels in machine learning where they exist, describe the importance of intrinsic dynamics for memory management and adaptation, and speculate on how disruption of extrinsic and intrinsic dynamics may give rise to mental disorders. Throughout, we also highlight algorithmic features of spine dynamics that may be relevant to future artificial intelligence developments.

New roles for dopamine in motor skill acquisition: lessons from primates, rodents, and songbirds

Motor learning is a core aspect of human life and appears to be ubiquitous throughout the animal kingdom. Dopamine, a neuromodulator with a multifaceted role in synaptic plasticity, may be a key signaling molecule for motor skill learning. Though typically studied in the context of reward-based associative learning, dopamine appears to be necessary for some types of motor learning. Mesencephalic dopamine structures are highly conserved among vertebrates, as are some of their primary targets within the basal ganglia, a subcortical circuit important for motor learning and motor control. With a focus on the benefits of cross-species comparisons, this review examines how "model-free" and "model-based" computational frameworks for understanding dopamine's role in associative learning may be applied to motor learning. The hypotheses that dopamine could drive motor learning either by functioning as a reward prediction error, through passive facilitating of normal basal ganglia activity, or through other mechanisms are examined in light of new studies using humans, rodents, and songbirds. Additionally, new paradigms that could enhance our understanding of dopamine's role in motor learning by bridging the gap between the theoretical literature on motor learning in humans and other species are discussed.