Opposite initialization to novel cues in dopamine signaling in ventral and posterior striatum in mice

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Functional diversity of dopamine axons in prefrontal cortex during classical conditioning

Midbrain dopamine neurons impact neural processing in the prefrontal cortex (PFC) through mesocortical projections. However, the signals conveyed by dopamine projections to the PFC remain unclear, particularly at the single-axon level. Here, we investigated dopaminergic axonal activity in the medial PFC (mPFC) during reward and aversive processing. By optimizing microprism-mediated two-photon calcium imaging of dopamine axon terminals, we found diverse activity in dopamine axons responsive to both reward and aversive stimuli. Some axons exhibited a preference for reward, while others favored aversive stimuli, and there was a strong bias for the latter at the population level. Long-term longitudinal imaging revealed that the preference was maintained in reward- and aversive-preferring axons throughout classical conditioning in which rewarding and aversive stimuli were paired with preceding auditory cues. However, as mice learned to discriminate reward or aversive cues, a cue activity preference gradually developed only in aversive-preferring axons. We inferred the trial-by-trial cue discrimination based on machine learning using anticipatory licking or facial expressions, and found that successful discrimination was accompanied by sharper selectivity for the aversive cue in aversive-preferring axons. Our findings indicate that a group of mesocortical dopamine axons encodes aversive-related signals, which are modulated by both classical conditioning across days and trial-by-trial discrimination within a day.

Cell-type specific auditory responses in the striatum are shaped by feed forward inhibition

The posterior "tail" region of the striatum receives dense innervation from sensory brain regions and has been demonstrated to play a role in behaviors that require sensorimotor integration including discrimination 1,2 , avoidance 3 and defense 4 responses. The output neurons of the striatum, the D1 and D2 striatal projection neurons (SPNs) that make up the direct and indirect pathways, respectively, are thought to play differential roles in these behavioral responses, although it remains unclear if or how these neurons display differential responsivity to sensory stimuli. Here, we used whole-cell recordings in vivo and ex vivo to examine the strength of excitatory and inhibitory synaptic inputs onto D1 and D2 SPNs following the stimulation of upstream auditory pathways. While D1 and D2 SPNs both displayed stimulus-evoked depolarizations, D1 SPN responses were stronger and faster for all stimuli tested in vivo as well as in brain slices. This difference did not arise from differences in the strength of excitatory inputs but from differences in the strength of feed forward inhibition. Indeed, fast spiking interneurons, which are readily engaged by auditory afferents exerted stronger inhibition onto D2 SPNs compared to D1 SPNs. Our results support a model in which differences in feed forward inhibition enable the preferential recruitment of the direct pathway in response to auditory stimuli, positioning this pathway to initiate sound-driven actions.

Dopamine and acetylcholine correlations in the nucleus accumbens depend on behavioral task states

Dopamine in the nucleus accumbens ramps up as animals approach desired goals. These ramps have received intense scrutiny because they seem to violate long-held hypotheses on dopamine function. Furthermore, it has been proposed that they are driven by local acetylcholine release, i.e., that they are mechanistically separate from dopamine signals related to reward prediction errors. Here, we tested this hypothesis by simultaneously recording accumbal dopamine and acetylcholine signals in rats executing a task involving motivated approach. Contrary to recent reports, we found that dopamine ramps were not coincidental with changes in acetylcholine. Instead, we found that acetylcholine could be positively, negatively, or uncorrelated with dopamine depending on whether the task phase was determined by a salient cue, reward prediction error, or active approach, respectively. Our results suggest that accumbal dopamine and acetylcholine are largely independent but may combine to engage different postsynaptic mechanisms depending on the behavioral task states.

Striatal dopamine signals reflect perceived cue-action-outcome associations in mice

Striatal dopamine drives associative learning by acting as a teaching signal. Much work has focused on simple learning paradigms, including Pavlovian and instrumental learning. However, higher cognition requires that animals generate internal concepts of their environment, where sensory stimuli, actions and outcomes become flexibly associated. Here, we performed fiber photometry dopamine measurements across the striatum of male mice as they learned cue-action-outcome associations based on implicit and changing task rules. Reinforcement learning models of the behavioral and dopamine data showed that rule changes lead to adjustments of learned cue-action-outcome associations. After rule changes, mice discarded learned associations and reset outcome expectations. Cue- and outcome-triggered dopamine signals became uncoupled and dependent on the adopted behavioral strategy. As mice learned the new association, coupling between cue- and outcome-triggered dopamine signals and task performance re-emerged. Our results suggest that dopaminergic reward prediction errors reflect an agent's perceived locus of control.

Targeted micro-fiber arrays for measuring and manipulating localized multi-scale neural dynamics over large, deep brain volumes during behavior

Neural population dynamics relevant to behavior vary over multiple spatial and temporal scales across three-dimensional volumes. Current optical approaches lack the spatial coverage and resolution necessary to measure and manipulate naturally occurring patterns of large-scale, distributed dynamics within and across deep brain regions such as the striatum. We designed a new micro-fiber array approach capable of chronically measuring and optogenetically manipulating local dynamics across over 100 targeted locations simultaneously in head-fixed and freely moving mice, enabling the investigation of cell-type- and neurotransmitter-specific signals over arbitrary 3D volumes at a spatial resolution and coverage previously inaccessible. We applied this method to resolve rapid dopamine release dynamics across the striatum, revealing distinct, modality-specific spatiotemporal patterns in response to salient sensory stimuli extending over millimeters of tissue. Targeted optogenetics enabled flexible control of neural signaling on multiple spatial scales, better matching endogenous signaling patterns, and the spatial localization of behavioral function across large circuits.

Glutamate inputs send prediction error of reward, but not negative value of aversive stimuli, to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), but the mechanisms underlying RPE computation, particularly the contributions of different neurotransmitters, remain poorly understood. Here, we used a genetically encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons in mice. We found that glutamate inputs exhibit virtually all of the characteristics of RPE rather than conveying a specific component of RPE computation, such as reward or expectation. Notably, whereas glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli into more positive responses, whereas excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Enhanced Punishment Responses in Patients With Schizophrenia: An Event-Related Potential Study

It is well known that abnormal reward processing is a characteristic feature of various psychopathologies including schizophrenia. Reduced reward anticipation has been suggested as a core symptom of schizophrenia. The Monetary Incentive Delay Task (MID) is frequently used to detect reward anticipation. The present study aims to evaluate the amplitude and latency of event-related potential (ERP) P300 in patients with schizophrenia (SCH) compared to healthy controls during the MID task. Twenty patients with SCH and 21 demographically matched healthy controls (HC) were included in the study. ERP P300 amplitude and latency values were compared between groups using an MID task in which reward and loss cues were presented. Relations between P300 and clinical facets were investigated in the patient group. SCH group had enhanced mean P300 amplitudes and delayed peak latency in the punishment condition compared with HC. These higher responses were also associated with negative symptoms. SCH group showed altered reward processing as being more sensitive to loss of reward conditions as firstly evidenced by electrophysiological methods, possibly due to abnormality in various systems including social withdrawal, social defeat, and behavioral inhibition system.

Curiosity: primate neural circuits for novelty and information seeking

For many years, neuroscientists have investigated the behavioural, computational and neurobiological mechanisms that support value-based decisions, revealing how humans and animals make choices to obtain rewards. However, many decisions are influenced by factors other than the value of physical rewards or second-order reinforcers (such as money). For instance, animals (including humans) frequently explore novel objects that have no intrinsic value solely because they are novel and they exhibit the desire to gain information to reduce their uncertainties about the future, even if this information cannot lead to reward or assist them in accomplishing upcoming tasks. In this Review, I discuss how circuits in the primate brain responsible for detecting, predicting and assessing novelty and uncertainty regulate behaviour and give rise to these behavioural components of curiosity. I also briefly discuss how curiosity-related behaviours arise during postnatal development and point out some important reasons for the persistence of curiosity across generations.

65 years of research on dopamine's role in classical fear conditioning and extinction: A systematic review

Dopamine, a catecholamine neurotransmitter, has historically been associated with the encoding of reward, whereas its role in aversion has received less attention. Here, we systematically gathered the vast evidence of the role of dopamine in the simplest forms of aversive learning: classical fear conditioning and extinction. In the past, crude methods were used to augment or inhibit dopamine to study its relationship with fear conditioning and extinction. More advanced techniques such as conditional genetic, chemogenic and optogenetic approaches now provide causal evidence for dopamine's role in these learning processes. Dopamine neurons encode conditioned stimuli during fear conditioning and extinction and convey the signal via activation of D1-4 receptor sites particularly in the amygdala, prefrontal cortex and striatum. The coordinated activation of dopamine receptors allows for the continuous formation, consolidation, retrieval and updating of fear and extinction memory in a dynamic and reciprocal manner. Based on the reviewed literature, we conclude that dopamine is crucial for the encoding of classical fear conditioning and extinction and contributes in a way that is comparable to its role in encoding reward.

Heterogeneity of mesencephalic dopaminergic neurons: From molecular classifications, electrophysiological properties to functional connectivity

The mesencephalic dopamine (DA) system is composed of neuronal subtypes that are molecularly and functionally distinct, are responsible for specific behaviors, and are closely associated with numerous brain disorders. Existing research has made significant advances in identifying the heterogeneity of mesencephalic DA neurons, which is necessary for understanding their diverse physiological functions and disease susceptibility. Moreover, there is a conflict regarding the electrophysiological properties of the distinct subsets of midbrain DA neurons. This review aimed to elucidate recent developments in the heterogeneity of midbrain DA neurons, including subpopulation categorization, electrophysiological characteristics, and functional connectivity to provide new strategies for accurately identifying distinct subtypes of midbrain DA neurons and investigating the underlying mechanisms of these neurons in various diseases.

The role of prospective contingency in the control of behavior and dopamine signals during associative learning

Associative learning depends on contingency, the degree to which a stimulus predicts an outcome. Despite its importance, the neural mechanisms linking contingency to behavior remain elusive. Here we examined the dopamine activity in the ventral striatum - a signal implicated in associative learning - in a Pavlovian contingency degradation task in mice. We show that both anticipatory licking and dopamine responses to a conditioned stimulus decreased when additional rewards were delivered uncued, but remained unchanged if additional rewards were cued. These results conflict with contingency-based accounts using a traditional definition of contingency or a novel causal learning model (ANCCR), but can be explained by temporal difference (TD) learning models equipped with an appropriate inter-trial-interval (ITI) state representation. Recurrent neural networks trained within a TD framework develop state representations like our best 'handcrafted' model. Our findings suggest that the TD error can be a measure that describes both contingency and dopaminergic activity.

A multi-stage anticipated surprise model with dynamic expectation for economic decision-making

There are many modeling works that aim to explain people's behaviors that violate classical economic theories. However, these models often do not take into full account the multi-stage nature of real-life problems and people's tendency in solving complicated problems sequentially. In this work, we propose a descriptive decision-making model for multi-stage problems with perceived post-decision information. In the model, decisions are chosen based on an entity which we call the 'anticipated surprise'. The reference point is determined by the expected value of the possible outcomes, which we assume to be dynamically changing during the mental simulation of a sequence of events. We illustrate how our formalism can help us understand prominent economic paradoxes and gambling behaviors that involve multi-stage or sequential planning. We also discuss how neuroscience findings, like prediction error signals and introspective neuronal replay, as well as psychological theories like affective forecasting, are related to the features in our model. This provides hints for future experiments to investigate the role of these entities in decision-making.

An opponent striatal circuit for distributional reinforcement learning

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.

Corticotropin-releasing factor-dopamine interactions in male and female macaque: Beyond the classic VTA

Dopamine (DA) is involved in stress and stress-related illnesses, including many psychiatric disorders. Corticotropin-releasing factor (CRF) plays a role in stress responses and targets the ventral midbrain DA system, which is composed of DA and non-DA cells, and divided into specific subregions. Although CRF inputs to the midline A10 nuclei ("classic VTA") are known, in monkeys, CRF-containing terminals are also highly enriched in the expanded A10 parabrachial pigmented nucleus (PBP) and in the A8 retrorubral field subregions. We characterized CRF-labeled synaptic terminals on DA (tyrosine hydroxylase, TH+) and non-DA (TH-) cell types in the PBP and A8 regions using immunoreactive electron microscopy (EM) in male and female macaques. CRF labeling was present mostly in axon terminals, which mainly contacted TH-negative dendrites in both subregions. Most CRF-positive terminals had symmetric profiles. In both PBP and A8, CRF symmetric (putative inhibitory) synapses onto TH-negative dendrites were significantly greater than asymmetric (putative excitatory) profiles. This overall pattern was similar in males and females, despite shifts in the size of these effects between regions depending on sex. Because stress and gonadal hormone shifts can influence CRF expression, we also did hormonal assays over a 6-month time period and found little variability in basal cortisol across similarly housed animals at the same age. Together our findings suggest that at baseline, CRF-positive synaptic terminals in the primate PBP and A8 are poised to regulate DA indirectly through synaptic contacts onto non-DA neurons.

Functional architecture of dopamine neurons driving fear extinction learning

The ability to extinguish fear responses to stimuli that no longer predict danger is critical for adaptive behavior and increases the likelihood of survival. During fear extinction, dopamine (DA) neurons signal the absence of the expected aversive outcome, and this extinction prediction error (EPE) signal is crucial for initiating and driving extinction learning. However, the neural circuits underlying the EPE signal have remained elusive. Here, we investigate the input-output circuitry of EPE-encoding DA neurons in male mice. By employing projection-specific fiber photometry and optogenetics, we demonstrate that these neurons project to a restricted subregion of the nucleus accumbens. Comprehensive anatomical analyses, as well as projection-specific chemogenetic manipulations combined with recordings of DA biosensors, further uncover the dorsal raphe as one key input structure critical for generating the EPE signal. Together, our results reveal for the first time the functional architecture of EPE-encoding DA neurons crucial for driving fear extinction learning.

Curiosity-driven exploration: foundations in neuroscience and computational modeling

Curiosity refers to the intrinsic desire of humans and animals to explore the unknown, even when there is no apparent reason to do so. Thus far, no single, widely accepted definition or framework for curiosity has emerged, but there is growing consensus that curious behavior is not goal-directed but related to seeking or reacting to information. In this review, we take a phenomenological approach and group behavioral and neurophysiological studies which meet these criteria into three categories according to the type of information seeking observed. We then review recent computational models of curiosity from the field of machine learning and discuss how they enable integrating different types of information seeking into one theoretical framework. Combinations of behavioral and neurophysiological studies along with computational modeling will be instrumental in demystifying the notion of curiosity.

The influence of dopamine autoreceptors on temperament and addiction risk

As a major regulator of dopamine (DA), DA autoreceptors (DAARs) exert substantial influence over DA-mediated behaviors. This paper reviews the physiological and behavioral impact of DAARs. Individual differences in DAAR functioning influences temperamental traits such as novelty responsivity and impulsivity, both of which are associated with vulnerability to addictive behavior in animal models and a broad array of externalizing behaviors in humans. DAARs additionally impact the response to psychostimulants and other drugs of abuse. Human PET studies of D2-like receptors in the midbrain provide evidence for parallels to the animal literature. These data lead to the proposal that weak DAAR regulation is a risk factor for addiction and externalizing problems. The review highlights the potential to build translational models of the functional role of DAARs in behavior. It also draws attention to key limitations in the current literature that would need to be addressed to further advance a weak DAAR regulation model of addiction and externalizing risk.

Targeted micro-fiber arrays for measuring and manipulating localized multi-scale neural dynamics over large, deep brain volumes during behavior

Neural population dynamics relevant for behavior vary over multiple spatial and temporal scales across 3-dimensional volumes. Current optical approaches lack the spatial coverage and resolution necessary to measure and manipulate naturally occurring patterns of large-scale, distributed dynamics within and across deep brain regions such as the striatum. We designed a new micro-fiber array and imaging approach capable of chronically measuring and optogenetically manipulating local dynamics across over 100 targeted locations simultaneously in head-fixed and freely moving mice. We developed a semi-automated micro-CT based strategy to precisely localize positions of each optical fiber. This highly-customizable approach enables investigation of multi-scale spatial and temporal patterns of cell-type and neurotransmitter specific signals over arbitrary 3-D volumes at a spatial resolution and coverage previously inaccessible. We applied this method to resolve rapid dopamine release dynamics across the striatum volume which revealed distinct, modality specific spatiotemporal patterns in response to salient sensory stimuli extending over millimeters of tissue. Targeted optogenetics through our fiber arrays enabled flexible control of neural signaling on multiple spatial scales, better matching endogenous signaling patterns, and spatial localization of behavioral function across large circuits.

Deep learning-based image analysis identifies a DAT-negative subpopulation of dopaminergic neurons in the lateral Substantia nigra

Here we present a deep learning-based image analysis platform (DLAP), tailored to autonomously quantify cell numbers, and fluorescence signals within cellular compartments, derived from RNAscope or immunohistochemistry. We utilised DLAP to analyse subtypes of tyrosine hydroxylase (TH)-positive dopaminergic midbrain neurons in mouse and human brain-sections. These neurons modulate complex behaviour, and are differentially affected in Parkinson's and other diseases. DLAP allows the analysis of large cell numbers, and facilitates the identification of small cellular subpopulations. Using DLAP, we identified a small subpopulation of TH-positive neurons (~5%), mainly located in the very lateral Substantia nigra (SN), that was immunofluorescence-negative for the plasmalemmal dopamine transporter (DAT), with ~40% smaller cell bodies. These neurons were negative for aldehyde dehydrogenase 1A1, with a lower co-expression rate for dopamine-D2-autoreceptors, but a ~7-fold higher likelihood of calbindin-d28k co-expression (~70%). These results have important implications, as DAT is crucial for dopamine signalling, and is commonly used as a marker for dopaminergic SN neurons.

Glutamate inputs send prediction error of reward but not negative value of aversive stimuli to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs) but the mechanisms underlying RPE computation, particularly contributions of different neurotransmitters, remain poorly understood. Here we used a genetically-encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons. We found that glutamate inputs exhibit virtually all of the characteristics of RPE, rather than conveying a specific component of RPE computation such as reward or expectation. Notably, while glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli toward more positive responses, while excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

CHL1 depletion affects dopamine receptor D2-dependent modulation of mouse behavior

Introduction

The dopaminergic system plays a key role in the appropriate functioning of the central nervous system, where it is essential for emotional balance, arousal, reward, and motor control. The cell adhesion molecule close homolog of L1 (CHL1) contributes to dopaminergic system development, and CHL1 and the dopamine receptor D2 (D2R) are associated with mental disorders like schizophrenia, addiction, autism spectrum disorder and depression.

Methods

Here, we investigated how the interplay between CHL1 and D2R affects the behavior of young adult male and female wild-type (CHL+/+) and CHL1-deficient (CHL1-/-) mice, when D2R agonist quinpirole and antagonist sulpiride are applied.

Results

Low doses of quinpirole (0.02 mg/kg body weight) induced hypolocomotion of CHL1+/+ and CHL1-/- males and females, but led to a delayed response in CHL1-/- mice. Sulpiride (1 mg/kg body weight) affected locomotion of CHL1-/- females and social interaction of CHL1+/+ females as well as social interactions of CHL1-/- and CHL1+/+ males. Quinpirole increased novelty-seeking behavior of CHL1-/- males compared to CHL1+/+ males. Vehicle-treated CHL1-/- males and females showed enhanced working memory and reduced stress-related behavior.

Discussion

We propose that CHL1 regulates D2R-dependent functions in vivo. Deficiency of CHL1 leads to abnormal locomotor activity and emotionality, and to sex-dependent behavioral differences.

Striatum supports fast learning but not memory recall

Animals learn to carry out motor actions in specific sensory contexts to achieve goals. The striatum has been implicated in producing sensory-motor associations, yet its contribution to memory formation or recall is not clear. To investigate the contribution of striatum to these processes, mice were taught to associate a cue, consisting of optogenetic activation of striatum-projecting neurons in visual cortex, with forelimb reaches to access food pellets. As necessary to direct learning, striatal neural activity encoded both the sensory context and outcome of reaching. With training, the rate of cued reaching increased, but brief optogenetic inhibition of striatal activity arrested learning and prevented trial-to-trial improvements in performance. However, the same manipulation did not affect performance improvements already consolidated into short- (within an hour) or long-term (across days) memories. Hence, striatal activity is necessary for trial-to-trial improvements in task performance, leading to plasticity in other brain areas that mediate memory recall.

Unique functional responses differentially map onto genetic subtypes of dopamine neurons

Dopamine neurons are characterized by their response to unexpected rewards, but they also fire during movement and aversive stimuli. Dopamine neuron diversity has been observed based on molecular expression profiles; however, whether different functions map onto such genetic subtypes remains unclear. In this study, we established that three genetic dopamine neuron subtypes within the substantia nigra pars compacta, characterized by the expression of Slc17a6 (Vglut2), Calb1 and Anxa1, each have a unique set of responses to rewards, aversive stimuli and accelerations and decelerations, and these signaling patterns are highly correlated between somas and axons within subtypes. Remarkably, reward responses were almost entirely absent in the Anxa1+ subtype, which instead displayed acceleration-correlated signaling. Our findings establish a connection between functional and genetic dopamine neuron subtypes and demonstrate that molecular expression patterns can serve as a common framework to dissect dopaminergic functions.

Moving, fast and slow: behavioural insights into bradykinesia in Parkinson's disease

The debilitating symptoms of Parkinson's disease, including the hallmark slowness of movement, termed bradykinesia, were described more than 100 years ago. Despite significant advances in elucidating the genetic, molecular and neurobiological changes in Parkinson's disease, it remains conceptually unclear exactly why patients with Parkinson's disease move slowly. To address this, we summarize behavioural observations of movement slowness in Parkinson's disease and discuss these findings in a behavioural framework of optimal control. In this framework, agents optimize the time it takes to gather and harvest rewards by adapting their movement vigour according to the reward that is at stake and the effort that needs to be expended. Thus, slow movements can be favourable when the reward is deemed unappealing or the movement very costly. While reduced reward sensitivity, which makes patients less inclined to work for reward, has been reported in Parkinson's disease, this appears to be related mainly to motivational deficits (apathy) rather than bradykinesia. Increased effort sensitivity has been proposed to underlie movement slowness in Parkinson's disease. However, careful behavioural observations of bradykinesia are inconsistent with abnormal computations of effort costs due to accuracy constraints or movement energetic expenditure. These inconsistencies can be resolved when considering that a general disability to switch between stable and dynamic movement states can contribute to an abnormal composite effort cost related to movement in Parkinson's disease. This can account for paradoxical observations such as the abnormally slow relaxation of isometric contractions or difficulties in halting a movement in Parkinson's disease, both of which increase movement energy expenditure. A sound understanding of the abnormal behavioural computations mediating motor impairment in Parkinson's disease will be vital for linking them to their underlying neural dynamics in distributed brain networks and for grounding future experimental studies in well-defined behavioural frameworks.

Sensory Reinforced Corticostriatal Plasticity

BACKGROUND

Regional changes in corticostriatal transmission induced by phasic dopaminergic signals are an essential feature of the neural network responsible for instrumental reinforcement during discovery of an action. However, the timing of signals that are thought to contribute to the induction of corticostriatal plasticity is difficult to reconcile within the framework of behavioural reinforcement learning, because the reinforcer is normally delayed relative to the selection and execution of causally-related actions.

OBJECTIVE

While recent studies have started to address the relevance of delayed reinforcement signals and their impact on corticostriatal processing, our objective was to establish a model in which a sensory reinforcer triggers appropriately delayed reinforcement signals relayed to the striatum via intact neuronal pathways and to investigate the effects on corticostriatal plasticity.

METHODS

We measured corticostriatal plasticity with electrophysiological recordings using a light flash as a natural sensory reinforcer, and pharmacological manipulations were applied in an in vivo anesthetized rat model preparation.

RESULTS

We demonstrate that the spiking of striatal neurons evoked by single-pulse stimulation of the motor cortex can be potentiated by a natural sensory reinforcer, operating through intact afferent pathways, with signal timing approximating that required for behavioural reinforcement. The pharmacological blockade of dopamine receptors attenuated the observed potentiation of corticostriatal neurotransmission.

CONCLUSION

This novel in vivo model of corticostriatal plasticity offers a behaviourally relevant framework to address the physiological, anatomical, cellular, and molecular bases of instrumental reinforcement learning.

Mesolimbic dopamine release precedes actively sought aversive stimuli in mice

In some models, animals approach aversive stimuli more than those housed in an enriched environment. Here, we found that male mice in an impoverished and unstimulating (i.e., boring) chamber without toys sought aversive air puffs more often than those in an enriched chamber. Using this animal model, we identified the insular cortex as a regulator of aversion-seeking behavior. Activation and inhibition of the insular cortex increased and decreased the frequencies of air-puff self-stimulation, respectively, and the firing patterns of insular neuron ensembles predicted the self-stimulation timing. Dopamine levels in the ventrolateral striatum decreased with passive air puffs but increased with actively sought puffs. Around 20% of mice developed intense self-stimulation despite being offered toys, which was prevented by administering opioid receptor antagonists. This study establishes a basis for comprehending the neural underpinnings of usually avoided stimulus-seeking behaviors.

Few-shot learning: temporal scaling in behavioral and dopaminergic learning

How do we learn associations in the world (e.g., between cues and rewards)? Cue-reward associative learning is controlled in the brain by mesolimbic dopamine1-4. It is widely believed that dopamine drives such learning by conveying a reward prediction error (RPE) in accordance with temporal difference reinforcement learning (TDRL) algorithms5. TDRL implementations are "trial-based": learning progresses sequentially across individual cue-outcome experiences. Accordingly, a foundational assumption-often considered a mere truism-is that the more cue-reward pairings one experiences, the more one learns this association. Here, we disprove this assumption, thereby falsifying a foundational principle of trial-based learning algorithms. Specifically, when a group of head-fixed mice received ten times fewer experiences over the same total time as another, a single experience produced as much learning as ten experiences in the other group. This quantitative scaling also holds for mesolimbic dopaminergic learning, with the increase in learning rate being so high that the group with fewer experiences exhibits dopaminergic learning in as few as four cue-reward experiences and behavioral learning in nine. An algorithm implementing reward-triggered retrospective learning explains these findings. The temporal scaling and few-shot learning observed here fundamentally changes our understanding of the neural algorithms of associative learning.

Development, wiring and function of dopamine neuron subtypes

The midbrain dopamine (mDA) system is composed of molecularly and functionally distinct neuron subtypes that mediate specific behaviours and are linked to various brain diseases. Considerable progress has been made in identifying mDA neuron subtypes, and recent work has begun to unveil how these neuronal subtypes develop and organize into functional brain structures. This progress is important for further understanding the disparate physiological functions of mDA neurons and their selective vulnerability in disease, and will ultimately accelerate therapy development. This Review discusses recent advances in our understanding of molecularly defined mDA neuron subtypes and their circuits, ranging from early developmental events, such as neuron migration and axon guidance, to their wiring and function, and future implications for therapeutic strategies.

Glutamatergic dysfunction leads to a hyper-dopaminergic phenotype through deficits in short-term habituation: a mechanism for aberrant salience

Psychosis in disorders like schizophrenia is commonly associated with aberrant salience and elevated striatal dopamine. However, the underlying cause(s) of this hyper-dopaminergic state remain elusive. Various lines of evidence point to glutamatergic dysfunction and impairments in synaptic plasticity in the etiology of schizophrenia, including deficits associated with the GluA1 AMPAR subunit. GluA1 knockout (Gria1^-/-) mice provide a model of impaired synaptic plasticity in schizophrenia and exhibit a selective deficit in a form of short-term memory which underlies short-term habituation. As such, these mice are unable to reduce attention to recently presented stimuli. In this study we used fast-scan cyclic voltammetry to measure phasic dopamine responses in the nucleus accumbens of Gria1^-/- mice to determine whether this behavioral phenotype might be a key driver of a hyper-dopaminergic state. There was no effect of GluA1 deletion on electrically-evoked dopamine responses in anaesthetized mice, demonstrating normal endogenous release properties of dopamine neurons in Gria1^-/- mice. Furthermore, dopamine signals were initially similar in Gria1^-/- mice compared to controls in response to both sucrose rewards and neutral light stimuli. They were also equally sensitive to changes in the magnitude of delivered rewards. In contrast, however, these stimulus-evoked dopamine signals failed to habituate with repeated presentations in Gria1^-/- mice, resulting in a task-relevant, hyper-dopaminergic phenotype. Thus, here we show that GluA1 dysfunction, resulting in impaired short-term habituation, is a key driver of enhanced striatal dopamine responses, which may be an important contributor to aberrant salience and psychosis in psychiatric disorders like schizophrenia.

Continuous cholinergic-dopaminergic updating in the nucleus accumbens underlies approaches to reward-predicting cues

The ability to learn Pavlovian associations from environmental cues predicting positive outcomes is critical for survival, motivating adaptive behaviours. This cued-motivated behaviour depends on the nucleus accumbens (NAc). NAc output activity mediated by spiny projecting neurons (SPNs) is regulated by dopamine, but also by cholinergic interneurons (CINs), which can release acetylcholine and glutamate via the activity of the vesicular acetylcholine transporter (VAChT) or the vesicular glutamate transporter (VGLUT3), respectively. Here we investigated behavioural and neurochemical changes in mice performing a touchscreen Pavlovian approach task by recording dopamine, acetylcholine, and calcium dynamics from D1- and D2-SPNs using fibre photometry in control, VAChT or VGLUT3 mutant mice to understand how these signals cooperate in the service of approach behaviours toward reward-predicting cues. We reveal that NAc acetylcholine-dopaminergic signalling is continuously updated to regulate striatal output underlying the acquisition of Pavlovian approach learning toward reward-predicting cues.

The zona incerta in control of novelty seeking and investigation across species

Many organisms rely on a capacity to rapidly replicate, disperse, and evolve when faced with uncertainty and novelty. But mammals do not evolve and replicate quickly. They rely on a sophisticated nervous system to generate predictions and select responses when confronted with these challenges. An important component of their behavioral repertoire is the adaptive context-dependent seeking or avoiding of perceptually novel objects, even when their values have not yet been learned. Here, we outline recent cross-species breakthroughs that shed light on how the zona incerta (ZI), a relatively evolutionarily conserved brain area, supports novelty-seeking and novelty-related investigations. We then conjecture how the architecture of the ZI's anatomical connectivity - the wide-ranging top-down cortical inputs to the ZI, and its specifically strong outputs to both the brainstem action controllers and to brain areas involved in action value learning - place the ZI in a unique role at the intersection of cognitive control and learning.

Striatal dopamine explains novelty-induced behavioral dynamics and individual variability in threat prediction

Animals both explore and avoid novel objects in the environment, but the neural mechanisms that underlie these behaviors and their dynamics remain uncharacterized. Here, we used multi-point tracking (DeepLabCut) and behavioral segmentation (MoSeq) to characterize the behavior of mice freely interacting with a novel object. Novelty elicits a characteristic sequence of behavior, starting with investigatory approach and culminating in object engagement or avoidance. Dopamine in the tail of the striatum (TS) suppresses engagement, and dopamine responses were predictive of individual variability in behavior. Behavioral dynamics and individual variability are explained by a reinforcement-learning (RL) model of threat prediction in which behavior arises from a novelty-induced initial threat prediction (akin to "shaping bonus") and a threat prediction that is learned through dopamine-mediated threat prediction errors. These results uncover an algorithmic similarity between reward- and threat-related dopamine sub-systems.

Nucleus accumbens dopamine tracks aversive stimulus duration and prediction but not value or prediction error

There is active debate on the role of dopamine in processing aversive stimuli, where inferred roles range from no involvement at all, to signaling an aversive prediction error (APE). Here, we systematically investigate dopamine release in the nucleus accumbens core (NAC), which is closely linked to reward prediction errors, in rats exposed to white noise (WN, a versatile, underutilized, aversive stimulus) and its predictive cues. Both induced a negative dopamine ramp, followed by slow signal recovery upon stimulus cessation. In contrast to reward conditioning, this dopamine signal was unaffected by WN value, context valence, or probabilistic contingencies, and the WN dopamine response shifted only partially toward its predictive cue. However, unpredicted WN provoked slower post-stimulus signal recovery than predicted WN. Despite differing signal qualities, dopamine responses to simultaneous presentation of rewarding and aversive stimuli were additive. Together, our findings demonstrate that instead of an APE, NAC dopamine primarily tracks prediction and duration of aversive events.

Dopaminergic circuits underlying associative aversive learning

Associative aversive learning enables animals to predict and avoid threats and thus is critical for survival and adaptive behavior. Anxiety disorders are characterized with deficits in normal aversive learning mechanisms and hence understanding the neural circuits underlying aversive learning and memory has high clinical relevance. Recent studies have revealed the dopamine system as one of the key modulators of aversive learning. In this review, we highlight recent advances that provide insights into how distinct dopaminergic circuits contribute to aversive learning and memory.

Behavioural and dopaminergic signatures of resilience

Chronic stress can have lasting adverse consequences in some individuals, yet others are resilient to the same stressor1,2. Susceptible and resilient individuals exhibit differences in the intrinsic properties of mesolimbic dopamine (DA) neurons after the stressful experience is over3-8. However, the causal links between DA, behaviour during stress and individual differences in resilience are unknown. Here we recorded behaviour in mice simultaneously with DA neuron activity in projections to the nucleus accumbens (NAc) (which signals reward9-12) and the tail striatum (TS) (which signals threat13-16) during social defeat. Supervised and unsupervised behavioural quantification revealed that during stress, resilient and susceptible mice use different behavioural strategies and have distinct activity patterns in DA terminals in the NAc (but not the TS). Neurally, resilient mice have greater activity near the aggressor, including at the onset of fighting back. Conversely, susceptible mice have greater activity at the offset of attacks and onset of fleeing. We also performed optogenetic stimulation of NAc-projecting DA neurons in open loop (randomly timed) during defeat or timed to specific behaviours using real-time behavioural classification. Both open-loop and fighting-back-timed activation promoted resilience and reorganized behaviour during defeat towards resilience-associated patterns. Together, these data provide a link between DA neural activity, resilience and resilience-associated behaviour during the experience of stress.

Salience memories formed by value, novelty and aversiveness jointly shape object responses in the prefrontal cortex and basal ganglia

Ecological fitness depends on maintaining object histories to guide future interactions. Recent evidence shows that value memory changes passive visual responses to objects in ventrolateral prefrontal cortex (vlPFC) and substantia nigra reticulata (SNr). However, it is not known whether this effect is limited to reward history and if not how cross-domain representations are organized within the same or different neural populations in this corticobasal circuitry. To address this issue, visual responses of the same neurons across appetitive, aversive and novelty domains were recorded in vlPFC and SNr. Results showed that changes in visual responses across domains happened in the same rather than separate populations and were related to salience rather than valence of objects. Furthermore, while SNr preferentially encoded outcome related salience memory, vlPFC encoded salience memory across all domains in a correlated fashion, consistent with its role as an information hub to guide behavior.

Laser stimulation of the skin for quantitative study of decision-making and motivation

Neuroeconomics studies how decision-making is guided by the value of rewards and punishments. But to date, little is known about how noxious experiences impact decisions. A challenge is the lack of an aversive stimulus that is dynamically adjustable in intensity and location, readily usable over many trials in a single experimental session, and compatible with multiple ways to measure neuronal activity. We show that skin laser stimulation used in human studies of aversion can be used for this purpose in several key animal models. We then use laser stimulation to study how neurons in the orbitofrontal cortex (OFC), an area whose many roles include guiding decisions among different rewards, encode the value of rewards and punishments. We show that some OFC neurons integrated the positive value of rewards with the negative value of aversive laser stimulation, suggesting that the OFC can play a role in more complex choices than previously appreciated.

Fiber photometry in striatum reflects primarily nonsomatic changes in calcium

Fiber photometry enables recording of population neuronal calcium dynamics in awake mice. While the popularity of fiber photometry has grown in recent years, it remains unclear whether photometry reflects changes in action potential firing (that is, 'spiking') or other changes in neuronal calcium. In microscope-based calcium imaging, optical and analytical approaches can help differentiate somatic from neuropil calcium. However, these approaches cannot be readily applied to fiber photometry. As such, it remains unclear whether the fiber photometry signal reflects changes in somatic calcium, changes in nonsomatic calcium or a combination of the two. Here, using simultaneous in vivo extracellular electrophysiology and fiber photometry, along with in vivo endoscopic one-photon and two-photon calcium imaging, we determined that the striatal fiber photometry does not reflect spiking-related changes in calcium and instead primarily reflects nonsomatic changes in calcium.

Neural signals implicated in the processing of appetitive and aversive events in social and non-social contexts

In 2014, we participated in a special issue of Frontiers examining the neural processing of appetitive and aversive events. Specifically, we reviewed brain areas that contribute to the encoding of prediction errors and value versus salience, attention and motivation. Further, we described how we disambiguated these cognitive processes and their neural substrates by using paradigms that incorporate both appetitive and aversive stimuli. We described a circuit in which the orbitofrontal cortex (OFC) signals expected value and the basolateral amygdala (BLA) encodes the salience and valence of both appetitive and aversive events. This information is integrated by the nucleus accumbens (NAc) and dopaminergic (DA) signaling in order to generate prediction and prediction error signals, which guide decision-making and learning via the dorsal striatum (DS). Lastly, the anterior cingulate cortex (ACC) is monitoring actions and outcomes, and signals the need to engage attentional control in order to optimize behavioral output. Here, we expand upon this framework, and review our recent work in which within-task manipulations of both appetitive and aversive stimuli allow us to uncover the neural processes that contribute to the detection of outcomes delivered to a conspecific and behaviors in social contexts. Specifically, we discuss the involvement of single-unit firing in the ACC and DA signals in the NAc during the processing of appetitive and aversive events in both social and non-social contexts.

Dopamine signaling in the nucleus accumbens core mediates latent inhibition

Studies investigating the neural mechanisms by which associations between cues and predicted outcomes control behavior often use associative learning frameworks to understand the neural control of behavior. These frameworks do not always account for the full range of effects that novelty can have on behavior and future associative learning. Here, in mice, we show that dopamine in the nucleus accumbens core is evoked by novel, neutral stimuli, and the trajectory of this response over time tracked habituation to these stimuli. Habituation to novel cues before associative learning reduced future associative learning, a phenomenon known as latent inhibition. Crucially, trial-by-trial dopamine response patterns tracked this phenomenon. Optogenetic manipulation of dopamine responses to the cue during the habituation period bidirectionally influenced future associative learning. Thus, dopamine signaling in the nucleus accumbens core has a causal role in novelty-based learning in a way that cannot be predicted based on purely associative factors.

A gradual temporal shift of dopamine responses mirrors the progression of temporal difference error in machine learning

A large body of evidence has indicated that the phasic responses of midbrain dopamine neurons show a remarkable similarity to a type of teaching signal (temporal difference (TD) error) used in machine learning. However, previous studies failed to observe a key prediction of this algorithm: that when an agent associates a cue and a reward that are separated in time, the timing of dopamine signals should gradually move backward in time from the time of the reward to the time of the cue over multiple trials. Here we demonstrate that such a gradual shift occurs both at the level of dopaminergic cellular activity and dopamine release in the ventral striatum in mice. Our results establish a long-sought link between dopaminergic activity and the TD learning algorithm, providing fundamental insights into how the brain associates cues and rewards that are separated in time.

Dopamine encodes real-time reward availability and transitions between reward availability states on different timescales

Optimal behavior requires interpreting environmental cues that indicate when to perform actions. Dopamine is important for learning about reward-predicting events, but its role in adapting to inhibitory cues is unclear. Here we show that when mice can earn rewards in the absence but not presence of an auditory cue, dopamine level in the ventral striatum accurately reflects reward availability in real-time over a sustained period (80 s). In addition, unpredictable transitions between different states of reward availability are accompanied by rapid (~1-2 s) dopamine transients that deflect negatively at the onset and positively at the offset of the cue. This Dopamine encoding of reward availability and transitions between reward availability states is not dependent on reward or activity evoked dopamine release, appears before mice learn the task and is sensitive to motivational state. Our findings are consistent across different techniques including electrochemical recordings and fiber photometry with genetically encoded optical sensors for calcium and dopamine.

Historical and Modern Evidence for the Role of Reward Circuitry in Emergence

Increasing evidence supports a role for brain reward circuitry in modulating arousal along with emergence from anesthesia. Emergence remains an important frontier for investigation, since no drug exists in clinical practice to initiate rapid and smooth emergence. This review discusses clinical and preclinical evidence indicating a role for two brain regions classically considered integral components of the mesolimbic brain reward circuitry, the ventral tegmental area and the nucleus accumbens, in emergence from propofol and volatile anesthesia. Then there is a description of modern systems neuroscience approaches to neural circuit investigations that will help span the large gap between preclinical and clinical investigation with the shared aim of developing therapies to promote rapid emergence without agitation or delirium. This article proposes that neuroscientists include models of whole-brain network activity in future studies to inform the translational value of preclinical investigations and foster productive dialogues with clinician anesthesiologists.

A unidirectional but not uniform striatal landscape of dopamine signaling for motivational stimuli

SignificanceAlthough it is undisputed that striatal dopamine plays a prominent role in motivated behavior and learning, the precise information conveyed by dopamine signals as such is under active debate. For a long time, the idea dominated that dopamine encodes a reward prediction error and that this signal is broadcast uniformly throughout the brain. However, here, we capture dopamine dynamics across many striatal regions and demonstrate that dopamine release is, regionally, extremely heterogeneous and that a reward prediction error-like signal is predominantly found in the relatively small limbic domain of the striatum. Another striking organizing principle is that stimulus valence directs dopamine concentration homogeneously across all regions (i.e., appetitive stimuli increase dopamine and aversive stimuli decrease dopamine).

The role of state uncertainty in the dynamics of dopamine

Reinforcement learning models of the basal ganglia map the phasic dopamine signal to reward prediction errors (RPEs). Conventional models assert that, when a stimulus predicts a reward with fixed delay, dopamine activity during the delay should converge to baseline through learning. However, recent studies have found that dopamine ramps up before reward in certain conditions even after learning, thus challenging the conventional models. In this work, we show that sensory feedback causes an unbiased learner to produce RPE ramps. Our model predicts that when feedback gradually decreases during a trial, dopamine activity should resemble a "bump," whose ramp-up phase should, furthermore, be greater than that of conditions where the feedback stays high. We trained mice on a virtual navigation task with varying brightness, and both predictions were empirically observed. In sum, our theoretical and experimental results reconcile the seemingly conflicting data on dopamine behaviors under the RPE hypothesis.

Conservation of the Direct and Indirect Pathway Dichotomy in Mouse Caudal Striatum With Uneven Distribution of Dopamine Receptor D1- and D2-Expressing Neurons

The striatum is one of the key nuclei for adequate control of voluntary behaviors and reinforcement learning. Two striatal projection neuron types, expressing either dopamine receptor D1 (D1R) or dopamine receptor D2 (D2R) constitute two independent output routes: the direct or indirect pathways, respectively. These pathways co-work in balance to achieve coordinated behavior. Two projection neuron types are equivalently intermingled in most striatal space. However, recent studies revealed two atypical zones in the caudal striatum: the zone in which D1R-neurons are the minor population (D1R-poor zone) and that in which D2R-neurons are the minority (D2R-poor zone). It remains obscure as to whether these imbalanced zones have similar properties on axonal projections and electrophysiology compared to other striatal regions. Based on morphological experiments in mice using immunofluorescence, in situ hybridization, and neural tracing, here, we revealed that the poor zones densely projected to the globus pallidus and substantia nigra pars lateralis, with a few collaterals in substantia nigra pars reticulata and compacta. Similar to that in other striatal regions, D1R-neurons were the direct pathway neurons. We also showed that the membrane properties of projection neurons in the poor zones were largely similar to those in the conventional striatum using in vitro electrophysiological recording. In addition, the poor zones existed irrespective of the age or sex of mice. We also identified the poor zones in the common marmoset as well as other rodents. These results suggest that the poor zones in the caudal striatum follow the conventional projection patterns irrespective of the imbalanced distribution of projection neurons. The poor zones could be an innate structure and common in mammals. The unique striatal zones possessing highly restricted projections could relate to functions different from those of motor-related striatum.

Seeking motivation and reward: roles of dopamine, hippocampus and supramammillo-septal pathway

Reinforcement learning and goal-seeking behavior are thought to be mediated by midbrain dopamine neurons. However, little is known about neural substrates of curiosity and exploratory behavior, which occur in the absence of clear goal or reward. This is despite behavioral scientists having long suggested that curiosity and exploratory behaviors are regulated by an innate drive. We refer to such behavior as information-seeking behavior and propose 1) key neural substrates and 2) the concept of environment prediction error as a framework to understand information-seeking processes. The cognitive aspect of information-seeking behavior, including the perception of salience and uncertainty, involves, in part, the pathways from the posterior hypothalamic supramammillary region to the hippocampal formation. The vigor of such behavior is modulated by the following: supramammillary glutamatergic neurons; their projections to medial septal glutamatergic neurons; and the projections of medial septal glutamatergic neurons to ventral tegmental dopaminergic neurons. Phasic responses of dopaminergic neurons are characterized as signaling potentially important stimuli rather than rewards. This paper describes how novel stimuli and uncertainty trigger seeking motivation and how these neural substrates modulate information-seeking behavior.

The prediction-error hypothesis of schizophrenia: new data point to circuit-specific changes in dopamine activity

Schizophrenia is a severe psychiatric disorder affecting 21 million people worldwide. People with schizophrenia suffer from symptoms including psychosis and delusions, apathy, anhedonia, and cognitive deficits. Strikingly, schizophrenia is characterised by a learning paradox involving difficulties learning from rewarding events, whilst simultaneously 'overlearning' about irrelevant or neutral information. While dysfunction in dopaminergic signalling has long been linked to the pathophysiology of schizophrenia, a cohesive framework that accounts for this learning paradox remains elusive. Recently, there has been an explosion of new research investigating how dopamine contributes to reinforcement learning, which illustrates that midbrain dopamine contributes in complex ways to reinforcement learning, not previously envisioned. This new data brings new possibilities for how dopamine signalling contributes to the symptomatology of schizophrenia. Building on recent work, we present a new neural framework for how we might envision specific dopamine circuits contributing to this learning paradox in schizophrenia in the context of models of reinforcement learning. Further, we discuss avenues of preclinical research with the use of cutting-edge neuroscience techniques where aspects of this model may be tested. Ultimately, it is hoped that this review will spur to action more research utilising specific reinforcement learning paradigms in preclinical models of schizophrenia, to reconcile seemingly disparate symptomatology and develop more efficient therapeutics.