Neural correlates of stimulus-response and response-outcome associations in dorsolateral versus dorsomedial striatum

Functional states of prelimbic and related circuits during the acquisition of a GO/noGO task in rats

GO/noGO tasks enable assessing decision-making processes and the ability to suppress a specific action according to the context. Here, rats had to discriminate between 2 visual stimuli (GO or noGO) shown on an iPad screen. The execution (for GO) or nonexecution (for noGO) of the selected action (to touch or not the visual display) were reinforced with food. The main goal was to record and to analyze local field potentials collected from cortical and subcortical structures when the visual stimuli were shown on the touch screen and during the subsequent activities. Rats were implanted with recording electrodes in the prelimbic cortex, primary motor cortex, nucleus accumbens septi, basolateral amygdala, dorsolateral and dorsomedial striatum, hippocampal CA1, and mediodorsal thalamic nucleus. Spectral analyses of the collected data demonstrate that the prelimbic cortex was selectively involved in the cognitive and motivational processing of the learning task but not in the execution of reward-directed behaviors. In addition, the other recorded structures presented specific tendencies to be involved in these 2 types of brain activity in response to the presentation of GO or noGO stimuli. Spectral analyses, spectrograms, and coherence between the recorded brain areas indicate their specific involvement in GO vs. noGO tasks.

Regulator of G Protein Signaling 14 protein expression profile in the adult mouse brain

Regulator of G protein signaling 14 (RGS14) is a multifunctional signaling protein that serves as a natural suppressor of synaptic plasticity in the mouse brain. Our previous studies showed that RGS14 is highly expressed in postsynaptic dendrites and spines of pyramidal neurons in hippocampal area CA2 of the developing mouse brain. However, our more recent work with adult rhesus macaque brain shows that RGS14 is found in multiple neuron populations throughout hippocampal area CA1 and CA2, caudate nucleus, putamen, globus pallidus, substantia nigra, and amygdala in the adult rhesus monkey brain. In the mouse brain, we also have observed RGS14 protein in discrete limbic regions linked to reward behavior and addiction, including the central amygdala and the nucleus accumbens, but a comprehensive mapping of RGS14 protein expression in the adult mouse brain is lacking. Here, we report that RGS14 is more broadly expressed in mouse brain than previously known. Intense RGS14 staining is observed in specific neuron populations of the hippocampal formation, amygdala, septum, bed nucleus of stria terminalis and ventral striatum/nucleus accumbens. RGS14 is also observed in axon fiber tracts including the dorsal fornix, fimbria, stria terminalis, and the ventrohippocampal commissure. Moderate RGS14 staining is observed in various other adjacent regions not previously reported. These findings show that RGS14 is expressed in brain regions that govern aspects of core cognitive functions such as sensory perception, emotion, memory, motivation, and execution of actions, and suggests that RGS14 may serve to suppress plasticity and filter inputs in these brain regions to set the overall tone on experience-to-action processes.

Chronic Ethanol Exposure Produces Persistent Impairment in Cognitive Flexibility and Decision Signals in the Striatum

Lack of cognitive flexibility is a hallmark of substance use disorders and has been associated with drug-induced synaptic plasticity in the dorsomedial striatum (DMS). Yet the possible impact of altered plasticity on real-time striatal neural dynamics during decision-making is unclear. Here, we identified persistent impairments induced by chronic ethanol (EtOH) exposure on cognitive flexibility and striatal decision signals. After a substantial withdrawal period from prior EtOH vapor exposure, male, but not female, rats exhibited reduced adaptability and exploratory behavior during a dynamic decision-making task. Reinforcement learning models showed that prior EtOH exposure enhanced learning from rewards over omissions. Notably, neural signals in the DMS related to the decision outcome were enhanced, while those related to choice and choice-outcome conjunction were reduced, in EtOH-treated rats compared to the controls. These findings highlight the profound impact of chronic EtOH exposure on adaptive decision-making, pinpointing specific changes in striatal representations of actions and outcomes as underlying mechanisms for cognitive deficits.

Detection of Spatially Localized Sounds Is Robust to Saccades and Concurrent Eye Movement-Related Eardrum Oscillations (EMREOs)

Hearing is an active process, and recent studies show that even the ear is affected by cognitive states or motor actions. One example are movements of the eardrum induced by saccadic eye movements, known as "eye movement-related eardrum oscillations" (EMREOs). While these are systematically shaped by the direction and size of saccades, the consequences of saccadic eye movements and their resulting EMREOs for hearing remain unclear. We here studied their implications for the detection of near-threshold clicks in human participants. Across three experiments, sound detection was not affected by their time of presentation relative to saccade onset, by saccade amplitude or direction. While the EMREOs were shaped by the direction and amplitude of the saccadic movement, inducing covert shifts in spatial attention did not affect the EMREO, suggesting that this signature of active sensing is restricted to overt changes in visual focus. Importantly, in our experiments, fluctuations in the EMREO amplitude were not related to detection performance, at least when monaural cues are sufficient. Hence, while eye movements may shape the transduction of acoustic information, the behavioral implications remain to be understood.SIGNIFICANCE STATEMENT Previous studies suggest that oculomotor behavior may influence how we perceive spatially localized sounds. Recent work has introduced a new perspective on this question by showing that eye movements can directly modulate the eardrum. Yet, it remains unclear whether this signature of active hearing accounts for behavioral effects. We here show that overt but not covert changes in visual attention modulate the eardrum, but these modulations do not interfere with the detection of sounds. Our results provide a starting point to obtain a deeper understanding about the interplay of oculomotor behavior and the active ear.

A multilayered bidirectional associative memory model for learning nonlinear tasks

A multilayered bidirectional associative memory neural network is proposed to account for learning nonlinear types of association. The model (denoted as the MF-BAM) is composed of two modules, the Multi-Feature extracting bidirectional associative memory (MF), which contains various unsupervised network layers, and a modified Bidirectional Associative Memory (BAM), which consists of a single supervised network layer. The MF generates successive feature patterns from the original inputs. These patterns change the relationship between the inputs and targets in a way that the BAM can learn. The model was tested on different nonlinear tasks, such as the N-bit, Double Moon and its variants, and the 3-class spiral task. Behaviors were reported through learning errors, decision zones, and recall performances. Results showed that it was possible to learn all tasks consistently. By manipulating the number of units per layer and the number of unsupervised network layers in the MF, it was possible to change the level of nonlinearity observed in the decision boundaries. Furthermore, results indicated that different behaviors were achieved from the same set of inputs by using the different generated patterns. These findings are significant as they showed how a BAM-inspired model could solve nonlinear tasks in a more cognitively plausible fashion.

Huntingtin recruits KIF1A to transport synaptic vesicle precursors along the mouse axon to support synaptic transmission and motor skill learning

Neurotransmitters are released at synapses by synaptic vesicles (SVs), which originate from SV precursors (SVPs) that have traveled along the axon. Because each synapse maintains a pool of SVs, only a small fraction of which are released, it has been thought that axonal transport of SVPs does not affect synaptic function. Here, studying the corticostriatal network both in microfluidic devices and in mice, we find that phosphorylation of the Huntingtin protein (HTT) increases axonal transport of SVPs and synaptic glutamate release by recruiting the kinesin motor KIF1A. In mice, constitutive HTT phosphorylation causes SV over-accumulation at synapses, increases the probability of SV release, and impairs motor skill learning on the rotating rod. Silencing KIF1A in these mice restored SV transport and motor skill learning to wild-type levels. Axonal SVP transport within the corticostriatal network thus influences synaptic plasticity and motor skill learning.

Multiplexed action-outcome representation by striatal striosome-matrix compartments detected with a mouse cost-benefit foraging task

Learning about positive and negative outcomes of actions is crucial for survival and underpinned by conserved circuits including the striatum. How associations between actions and outcomes are formed is not fully understood, particularly when the outcomes have mixed positive and negative features. We developed a novel foraging (‘bandit’) task requiring mice to maximize rewards while minimizing punishments. By 2-photon Ca⁺⁺ imaging, we monitored activity of visually identified anterodorsal striatal striosomal and matrix neurons. We found that action-outcome associations for reward and punishment were encoded in parallel in partially overlapping populations. Single neurons could, for one action, encode outcomes of opposing valence. Striosome compartments consistently exhibited stronger representations of reinforcement outcomes than matrix, especially for high reward or punishment prediction errors. These findings demonstrate multiplexing of action-outcome contingencies by single identified striatal neurons and suggest that striosomal neurons are particularly important in action-outcome learning.

The role that the striatum plays in tracking the association between actions and combinations of rewarding and aversive outcomes remains unclear. Here, by using both calcium imaging in mice and reinforcement learning models, the authors find that individual striatal neurons can encode associations between actions and multiple, sometimes conflicting, outcomes.

Striatum expresses region-specific plasticity consistent with distinct memory abilities

The striatum mediates two learning modalities: goal-directed behavior in dorsomedial (DMS) and habits in dorsolateral (DLS) striata. The synaptic bases of these learnings are still elusive. Indeed, while ample research has described DLS plasticity, little remains known about DMS plasticity and its involvement in procedural learning. Here, we find symmetric and asymmetric anti-Hebbian spike-timing-dependent plasticity (STDP) in DMS and DLS, respectively, with opposite plasticity dominance upon increasing corticostriatal activity. During motor-skill learning, plasticity is engaged in DMS and striatonigral DLS neurons only during early learning stages, whereas striatopallidal DLS neurons are mobilized only during late phases. With a mathematical modeling approach, we find that symmetric anti-Hebbian STDP favors memory flexibility, while asymmetric anti-Hebbian STDP favors memory maintenance, consistent with memory processes at play in procedural learning.

Maintained goal-directed control with overtraining on ratio schedules

It is thought that goal-directed control of actions weakens or becomes masked by habits over time. We tested the opposing hypothesis that goal-directed control becomes stronger over time, and that this growth is modulated by the overall action-outcome contiguity. Despite group differences in action-outcome contiguity early in training, rats trained under random and fixed ratio schedules showed equivalent goal-directed control of lever pressing that appeared to grow over time. We confirmed that goal-directed control was maintained after extended training under another type of ratio schedule-continuous reinforcement-using specific satiety and taste aversion devaluation methods. These results add to the growing literature showing that extensive training does not reliably weaken goal-directed control and that it may strengthen it, or at least maintain it.

The learning of prospective and retrospective cognitive maps within neural circuits

Brain circuits are thought to form a "cognitive map" to process and store statistical relationships in the environment. A cognitive map is commonly defined as a mental representation that describes environmental states (i.e., variables or events) and the relationship between these states. This process is commonly conceptualized as a prospective process, as it is based on the relationships between states in chronological order (e.g., does reward follow a given state?). In this perspective, we expand this concept on the basis of recent findings to postulate that in addition to a prospective map, the brain forms and uses a retrospective cognitive map (e.g., does a given state precede reward?). In doing so, we demonstrate that many neural signals and behaviors (e.g., habits) that seem inflexible and non-cognitive can result from retrospective cognitive maps. Together, we present a significant conceptual reframing of the neurobiological study of associative learning, memory, and decision making.

Dorsolateral Striatal Task-initiation Bursts Represent Past Experiences More than Future Action Plans

The dorsolateral striatum (DLS) is involved in learning and executing procedural actions. Cell ensembles in the DLS, but not the dorsomedial striatum (DMS), exhibit a burst of firing at the start of a well-learned action sequence ("task-bracketing"). However, it is currently unclear what information is contained in these bursts. Some theories suggest that these bursts should represent the procedural action sequence itself (that they should be about future action chains), whereas others suggest that they should contain representations of the current state of the world, taking into account primarily past information. In addition, the DLS local field potential shows transient bursts of power in the 50 Hz range (γ50) around the time a learned action sequence is initiated. However, it is currently unknown how bursts of activity in DLS cell ensembles and bursts of γ50 power in the DLS local field potential are related to each other. We found that DLS bursts at lap initiation in rats represented recently experienced reward locations more than future procedural actions, indicating that task-initiation DLS bursts contain primarily retrospective, rather than prospective, information to guide procedural actions. Furthermore, representations of past reward locations increased during periods of increased γ50 power in the DLS. There was no evidence of task-initiation bursts, increased γ50 power, or retrospective reward location information in the neighboring dorsomedial striatum. These data support a role for the DLS in model-free theories of procedural decision-making over planned action-chain theories, suggesting that procedural actions derive from representations of the current and recent past.SIGNIFICANCE STATEMENT While it is well-established that the dorsolateral striatum (DLS) plays a critical role in procedural decision-making, open questions remain about the kinds of representations contained in DLS ensemble activity that guide procedural actions. We found that DLS, but not DMS, cell ensembles contained nonlocal representations of past reward locations that appear moments before task-initiation DLS bursts. These retrospective representations were temporally linked to a rise in γ50 power that also preceded the characteristic DLS burst at task-initiation. These results support models of procedural decision-making based on associations between available actions and the current state of the world over models based on planning over action-chains.

Opposing roles for striatonigral and striatopallidal neurons in dorsolateral striatum in consolidating new instrumental actions

Comparatively little is known about how new instrumental actions are encoded in the brain. Using whole-brain c-Fos mapping, we show that neural activity is increased in the anterior dorsolateral striatum (aDLS) of mice that successfully learn a new lever-press response to earn food rewards. Post-learning chemogenetic inhibition of aDLS disrupts consolidation of the new instrumental response. Similarly, post-learning infusion of the protein synthesis inhibitor anisomycin into the aDLS disrupts consolidation of the new response. Activity of D1 receptor-expressing medium spiny neurons (D1-MSNs) increases and D2-MSNs activity decreases in the aDLS during consolidation. Chemogenetic inhibition of D1-MSNs in aDLS disrupts the consolidation process whereas D2-MSN inhibition strengthens consolidation but blocks the expression of previously learned habit-like responses. These findings suggest that D1-MSNs in the aDLS encode new instrumental actions whereas D2-MSNs oppose this new learning and instead promote expression of habitual actions.

Complete representation of action space and value in all dorsal striatal pathways

The dorsal striatum plays a central role in the selection, execution, and evaluation of actions. An emerging model attributes action selection to the matrix and evaluation to the striosome compartment. Here, we use large-scale cell-type-specific calcium imaging to determine the activity of striatal projection neurons (SPNs) during motor and decision behaviors in the three major outputs of the dorsomedial striatum: Oprm1⁺ striosome versus D1⁺ direct and A2A⁺ indirect pathway SPNs. We find that Oprm1⁺ SPNs show complex tunings to simple movements and value-guided actions, which are conserved across many sessions in a single task but remap between contexts. During decision making, the SPN tuning profiles form a complete representation in which sequential SPN activity jointly encodes task progress and value. We propose that the three major output pathways in the dorsomedial striatum share a similarly complete representation of the entire action space, including task- and phase-specific signals of action value and choice.

Acetylcholine from the nucleus basalis magnocellularis facilitates the retrieval of well-established memory

Retrieval deficit of long-term memory is a cardinal symptom of dementia and has been proposed to associate with abnormalities in the central cholinergic system. Difficulty in the retrieval of memory is experienced by healthy individuals and not limited to patients with neurological disorders that result in forgetfulness. The difficulty of retrieving memories is associated with various factors, such as how often the event was experienced or remembered, but it is unclear how the cholinergic system plays a role in the retrieval of memory formed by a daily routine (accumulated experience). To investigate this point, we trained rats moderately (for a week) or extensively (for a month) to detect a visual cue in a two-alternative forced-choice task. First, we confirmed the well-established memory in the extensively trained group was more resistant to the retrieval problem than recently acquired memory in the moderately trained group. Next, we tested the effect of a cholinesterase inhibitor, donepezil, on the retrieval of memory after a long no-task period in extensively trained rats. Pre-administration of donepezil improved performance and reduced the latency of task initiation compared to the saline-treated group. Finally, we lesioned cholinergic neurons of the nucleus basalis magnocellularis (NBM), which project to the entire neocortex, by injecting the cholinergic toxin 192 IgG-saporin. NBM-lesioned rats showed severely impaired task initiation and performance. These abilities recovered as the trials progressed, though they never reached the level observed in rats with intact NBM. These results suggest that acetylcholine released from the NBM contributes to the retrieval of well-established memory developed by a daily routine.

Depression in the Direct Pathway of the Dorsomedial Striatum Permits the Formation of Habitual Action

In order to achieve optimal outcomes in an ever-changing environment, humans and animals generally manage their action control via either goal-directed action or habitual action. These two action strategies are thought to be encoded in distinct parallel circuits in the dorsal striatum, specifically, the posterior dorsomedial striatum (DMS) and the dorsolateral striatum (DLS), respectively. The striatum is primarily composed of two subtypes of medium spiny neurons (MSNs): the direct-pathway striatonigral and the indirect-pathway striatopallidal MSNs. MSN-subtype-specific synaptic plasticity in the DMS and the DLS has been revealed to underlie goal-directed action and habitual action, respectively. However, whether any MSN-subtype-specific synaptic plasticity in the DMS is associated with habitual action, and if so, whether the synaptic plasticity affects the formation of habitual action, are not known. This study demonstrates that postsynaptic depression in the excitatory synapses of the direct-pathway striatonigral MSNs in the DMS is formed after habit learning. Moreover, chemogenetically rescuing this depression compromises the acquisition, but not the expression, of habitual action. These findings reveal that an MSN-subtype-specific synaptic plasticity in the DMS affects habitual action and suggest that plasticity in the DMS as well as in the DLS contributes to the formation of habitual action.

Orbitofrontal State Representations Are Related to Choice Adaptations and Reward Predictions

Animals can categorize the environment into "states," defined by unique sets of available action-outcome contingencies in different contexts. Doing so helps them choose appropriate actions and make accurate outcome predictions when in each given state. State maps have been hypothesized to be held in the orbitofrontal cortex (OFC), an area implicated in decision-making and encoding information about outcome predictions. Here we recorded neural activity in OFC in 6 male rats to test state representations. Rats were trained on an odor-guided choice task consisting of five trial blocks containing distinct sets of action-outcome contingencies, constituting states, with unsignaled transitions between them. OFC neural ensembles were analyzed using decoding algorithms. Results indicate that the vast majority of OFC neurons contributed to representations of the current state at any point in time, independent of odor cues and reward delivery, even at the level of individual neurons. Across state transitions, these representations gradually integrated evidence for the new state; the rate at which this integration happened in the prechoice part of the trial was related to how quickly the rats' choices adapted to the new state. Finally, OFC representations of outcome predictions, often thought to be the primary function of OFC, were dependent on the accuracy of OFC state representations.SIGNIFICANCE STATEMENT A prominent hypothesis proposes that orbitofrontal cortex (OFC) tracks current location in a "cognitive map" of state space. Here we tested this idea in detail by analyzing neural activity recorded in OFC of rats performing a task consisting of a series of states, each defined by a set of available action-outcome contingencies. Results show that most OFC neurons contribute to state representations and that these representations are related to the rats' decision-making and OFC reward predictions. These findings suggest new interpretations of emotional dysregulation in pathologies, such as addiction, which have long been known to be related to OFC dysfunction.

Discovering the Lost Reward: Critical Locations for Endocannabinoid Modulation of the Cortico-Striatal Loop That Are Implicated in Major Depression

Depression, the most prevalent psychiatric disorder in the Western world, is characterized by increased negative affect (i.e., depressed mood, cost value increase) and reduced positive affect (i.e., anhedonia, reward value decrease), fatigue, loss of appetite, and reduced psychomotor activity except for cases of agitative depression. Some forms, such as post-partum depression, have a high risk for suicidal attempts. Recent studies in humans and in animal models relate major depression occurrence and reoccurrence to alterations in dopaminergic activity, in addition to other neurotransmitter systems. Imaging studies detected decreased activity in the brain reward circuits in major depression. Therefore, the location of dopamine receptors in these circuits is relevant for understanding major depression. Interestingly, in cortico-striatal-dopaminergic pathways within the reward and cost circuits, the expression of dopamine and its contribution to reward are modulated by endocannabinoid receptors. These receptors are enriched in the striosomal compartment of striatum that selectively projects to dopaminergic neurons of substantia nigra compacta and is vulnerable to stress. This review aims to show the crosstalk between endocannabinoid and dopamine receptors and their vulnerability to stress in the reward circuits, especially in corticostriatal regions. The implications for novel treatments of major depression are discussed.

Whole-Brain Mapping of Direct Inputs to Dopamine D1 and D2 Receptor-Expressing Medium Spiny Neurons in the Posterior Dorsomedial Striatum

The posterior dorsomedial striatum (pDMS) is mainly composed of medium spiny neurons (MSNs) expressing either dopamine D1 receptors (D1Rs) or D2Rs. Activation of these two MSN types produces opposing effects on addictive behaviors. However, it remains unclear whether pDMS D1-MSNs or D2-MSNs receive afferent inputs from different brain regions or whether the extrastriatal afferents express distinct dopamine receptors. To assess whether these afferents also contained D1Rs or D2Rs, we generated double transgenic mice, in which D1R-expressing and D2R-expressing neurons were fluorescently labeled. We used rabies virus-mediated retrograde tracing in these mice to perform whole-brain mapping of direct inputs to D1-MSNs or D2-MSNs in the pDMS. We found that D1-MSNs preferentially received inputs from the secondary motor, secondary visual, and cingulate cortices, whereas D2-MSNs received inputs from the primary motor and primary sensory cortices, and the thalamus. We also discovered that the bed nucleus of the stria terminalis (BNST) and the central nucleus of the amygdala (CeA) contained abundant D2R-expressing, but few D1R-expressing, neurons in a triple transgenic mouse model. Remarkably, although limited D1R or D2R expression was observed in extrastriatal neurons that projected to D1-MSNs or D2-MSNs, we found that cortical structures preferentially contained D1R-expressing neurons that projected to D1-MSNs or D2-MSNs, while the thalamus, substantia nigra pars compacta (SNc), and BNST had more D2R-expressing cells that projected to D2-MSNs. Taken together, these findings provide a foundation for future understanding of the pDMS circuit and its role in action selection and reward-based behaviors.

Lessons From Deep Neural Networks for Studying the Coding Principles of Biological Neural Networks

One of the central goals in systems neuroscience is to understand how information is encoded in the brain, and the standard approach is to identify the relation between a stimulus and a neural response. However, the feature of a stimulus is typically defined by the researcher's hypothesis, which may cause biases in the research conclusion. To demonstrate potential biases, we simulate four likely scenarios using deep neural networks trained on the image classification dataset CIFAR-10 and demonstrate the possibility of selecting suboptimal/irrelevant features or overestimating the network feature representation/noise correlation. Additionally, we present studies investigating neural coding principles in biological neural networks to which our points can be applied. This study aims to not only highlight the importance of careful assumptions and interpretations regarding the neural response to stimulus features but also suggest that the comparative study between deep and biological neural networks from the perspective of machine learning can be an effective strategy for understanding the coding principles of the brain.

Prior Cocaine Use Alters the Normal Evolution of Information Coding in Striatal Ensembles during Value-Guided Decision-Making

Substance use disorders (SUDs) are characterized by maladaptive behavior. The ability to properly adjust behavior according to changes in environmental contingencies necessitates the interlacing of existing memories with updated information. This can be achieved by assigning learning in different contexts to compartmentalized "states." Though not often framed this way, the maladaptive behavior observed in individuals with SUDs may result from a failure to properly encode states because of drug-induced neural alterations. Previous studies found that the dorsomedial striatum (DMS) is important for behavioral flexibility and state encoding, suggesting the DMS may be an important substrate for these effects. Here, we recorded DMS neural activity in cocaine-experienced male rats during a decision-making task where blocks of trials represented distinct states to probe whether the encoding of state and state-related information is affected by prior drug exposure. We found that DMS medium spiny neurons (MSNs) and fast-spiking interneurons (FSIs) encoded such information and that prior cocaine experience disrupted the evolution of representations both within trials and across recording sessions. Specifically, DMS MSNs and FSIs from cocaine-experienced rats demonstrated higher classification accuracy of trial-specific rules, defined by response direction and value, compared with those drawn from sucrose-experienced rats, and these overly strengthened trial-type representations were related to slower switching behavior and reaction times. These data show that prior cocaine experience paradoxically increases the encoding of state-specific information and rules in the DMS and suggest a model in which abnormally specific and persistent representation of rules throughout trials in DMS slows value-based decision-making in well trained subjects.SIGNIFICANCE STATEMENT Substance use disorders (SUDs) may result from a failure to properly encode rules guiding situationally appropriate behavior. The dorsomedial striatum (DMS) is thought to be important for such behavioral flexibility and encoding that defines the situation or "state." This suggests that the DMS may be an important substrate for the maladaptive behavior observed in SUDs. In the current study, we show that prior cocaine experience results in over-encoding of state-specific information and rules in the DMS, which may impair normal adaptive decision-making in the task, akin to what is observed in SUDs.

The Hippocampus and Dorsolateral Striatum Integrate Distinct Types of Memories through Time and Space, Respectively

Several decades of research have established that different kinds of memories result from the activity of discrete neural networks. Studying how these networks process information in experiments that target specific types of mnemonic representations has provided deep insights into memory architecture and its neural underpinnings. However, in natural settings reality confronts organisms with problems that are not neatly compartmentalized. Thus, a critical problem in memory research that still needs to be addressed is how distinct types of memories are ultimately integrated. Here we demonstrate how two memory networks, the hippocampus and dorsolateral striatum, may accomplish such a goal. The hippocampus supports memory for facts and events, collectively known as declarative memory and often studied as spatial memory in rodents. The dorsolateral striatum provides the basis for habits that are assessed in stimulus-response types of tasks. Expanding previous findings, the current work revealed that in male Long-Evans rats, the hippocampus and dorsolateral striatum use time and space in distinct and largely complementary ways to integrate spatial and habitual representations. Specifically, the hippocampus supported both types of memories when they were formed in temporal juxtaposition, even if the learning took place in different environments. In contrast, the lateral striatum supported both types of memories if they were formed in the same environment, even at temporally distinct points. These results reveal for the first time that by using fundamental aspects of experience in specific ways, the hippocampus and dorsolateral striatum can transcend their attributed roles in information storage.SIGNIFICANCE STATEMENT The current paradigm in memory research postulates that different types of memories reflected in separate types of behavioral strategies result from activity in distinct neural circuits. However, recent data have shown that when rats concurrently acquired in the same environment of hippocampal-dependent spatial navigation and striatal-dependent approach of a visual cue, each of the two types of memories became dependent on both the hippocampus and dorsolateral striatum. The current work reveals that the hippocampus and dorsolateral striatum use distinct and complementary principles to integrate different types of memories in time and space: the hippocampus integrates memories formed in temporal proximity, while the lateral striatum integrates memories formed in the same space.

Striosomes Mediate Value-Based Learning Vulnerable in Age and a Huntington's Disease Model

Learning valence-based responses to favorable and unfavorable options requires judgments of the relative value of the options, a process necessary for species survival. We found, using engineered mice, that circuit connectivity and function of the striosome compartment of the striatum are critical for this type of learning. Calcium imaging during valence-based learning exhibited a selective correlation between learning and striosomal but not matrix signals. This striosomal activity encoded discrimination learning and was correlated with task engagement, which, in turn, could be regulated by chemogenetic excitation and inhibition. Striosomal function during discrimination learning was disturbed with aging and severely so in a mouse model of Huntington's disease. Anatomical and functional connectivity of parvalbumin-positive, putative fast-spiking interneurons (FSIs) to striatal projection neurons was enhanced in striosomes compared with matrix in mice that learned. Computational modeling of these findings suggests that FSIs can modulate the striosomal signal-to-noise ratio, crucial for discrimination and learning.

Redefining the relationship between effort and reward: Choice-execution model of effort-based decisions

Neuroscientific studies reliably demonstrate that rewards play a crucial role in guiding our choices when confronted with different effortful actions we could make. At the same time, psychological and economic research shows that effort we exert is not reliably predicted by the rewards we end up receiving. Why the mismatch between the two lines of evidence? Inspired by neuroscientific literature, we argue that value-based models of decision-making expose the complexity of the relationship between effort and reward, which changes between two crucial stages of the effort-based decision making process: Choice (i.e. action selection) and Execution (i.e. action execution involving actual effort exertion). To test this assumption, in the present study we set up two experiments (E1: N = 72, E2: N = 87), using a typical neuroscientific effort-based decision-making task. The findings of these experiments reveal that when making prospective choices, rewards do guide the level of effort people are prepared to exert, consistent with typical findings from Neuroscience. At a later stage, during execution of effortful actions, performance is determined by the actual amount of effort that needs to be exerted, consistent with psychological and behavioral economic research. We use the model we tested and the findings we generated to highlight critical new insights into effort-reward relationship, bringing different literatures together in the context of questions regarding what effort its, and the role that values play.

No substantial change in the balance between model-free and model-based control via training on the two-step task

Human decisions can be habitual or goal-directed, also known as model-free (MF) or model-based (MB) control. Previous work suggests that the balance between the two decision systems is impaired in psychiatric disorders such as compulsion and addiction, via overreliance on MF control. However, little is known whether the balance can be altered through task training. Here, 20 healthy participants performed a well-established two-step task that differentiates MB from MF control, across five training sessions. We used computational modelling and functional near-infrared spectroscopy to assess changes in decision-making and brain hemodynamic over time. Mixed-effects modelling revealed overall no substantial changes in MF and MB behavior across training. Although our behavioral and brain findings show task-induced changes in learning rates, these parameters have no direct relation to either MF or MB control or the balance between the two systems, and thus do not support the assumption of training effects on MF or MB strategies. Our findings indicate that training on the two-step paradigm in its current form does not support a shift in the balance between MF and MB control. We discuss these results with respect to implications for restoring the balance between MF and MB control in psychiatric conditions.

Psychiatric conditions such as compulsion or addiction are associated with an overreliance on habitual, or model-free, decision-making. Goal-directed, or model-based, decision-making may protect against such overreliance. We therefore asked whether model-free control could be reduced, and model-based control strengthened, via task training. We used the well-characterized two-step task that differentiates model-based from model-free actions. Our results suggest that training on the current form of the two-step task does not support a shift in the balance between model-free and model-based strategies. Factors such as devaluation, demotivation or automatization during training may play a role in the missing emergence of a training effect. Future studies could adapt the two-step task so as to separate such factors from decision-making strategies.

Learning, memory and consolidation mechanisms for behavioral control in hierarchically organized cortico-basal ganglia systems

This article aims to provide a synthesis on the question how brain structures cooperate to accomplish hierarchically organized behaviors, characterized by low‐level, habitual routines nested in larger sequences of planned, goal‐directed behavior. The functioning of a connected set of brain structures—prefrontal cortex, hippocampus, striatum, and dopaminergic mesencephalon—is reviewed in relation to two important distinctions: (a) goal‐directed as opposed to habitual behavior and (b) model‐based and model‐free learning. Recent evidence indicates that the orbitomedial prefrontal cortices not only subserve goal‐directed behavior and model‐based learning, but also code the “landscape” (task space) of behaviorally relevant variables. While the hippocampus stands out for its role in coding and memorizing world state representations, it is argued to function in model‐based learning but is not required for coding of action–outcome contingencies, illustrating that goal‐directed behavior is not congruent with model‐based learning. While the dorsolateral and dorsomedial striatum largely conform to the dichotomy between habitual versus goal‐directed behavior, ventral striatal functions go beyond this distinction. Next, we contextualize findings on coding of reward‐prediction errors by ventral tegmental dopamine neurons to suggest a broader role of mesencephalic dopamine cells, viz. in behavioral reactivity and signaling unexpected sensory changes. We hypothesize that goal‐directed behavior is hierarchically organized in interconnected cortico‐basal ganglia loops, where a limbic‐affective prefrontal‐ventral striatal loop controls action selection in a dorsomedial prefrontal–striatal loop, which in turn regulates activity in sensorimotor‐dorsolateral striatal circuits. This structure for behavioral organization requires alignment with mechanisms for memory formation and consolidation. We propose that frontal corticothalamic circuits form a high‐level loop for memory processing that initiates and temporally organizes nested activities in lower‐level loops, including the hippocampus and the ripple‐associated replay it generates. The evidence on hierarchically organized behavior converges with that on consolidation mechanisms in suggesting a frontal‐to‐caudal directionality in processing control.

Stable Representations of Decision Variables for Flexible Behavior

Decisions occur in dynamic environments. In the framework of reinforcement learning, the probability of performing an action is influenced by decision variables. Discrepancies between predicted and obtained rewards (reward prediction errors) update these variables, but they are otherwise stable between decisions. Although reward prediction errors have been mapped to midbrain dopamine neurons, it is unclear how the brain represents decision variables themselves. We trained mice on a dynamic foraging task in which they chose between alternatives that delivered reward with changing probabilities. Neurons in the medial prefrontal cortex, including projections to the dorsomedial striatum, maintained persistent firing rate changes over long timescales. These changes stably represented relative action values (to bias choices) and total action values (to bias response times) with slow decay. In contrast, decision variables were weakly represented in the anterolateral motor cortex, a region necessary for generating choices. Thus, we define a stable neural mechanism to drive flexible behavior.

Striatal dopamine and the temporal control of behavior

Striatal dopamine strongly regulates how individuals use time to guide behavior. Dopamine acts on D1- and D2- dopamine receptors in the striatum. However, the relative role of these receptors in the temporal control of behavior is unclear. To assess this, we trained rats on a task in which they decided to start and stop a series of responses based on the passage of time and evaluated how blocking D1 or D2-dopamine receptors in the dorsomedial or dorsolateral striatum impacted performance. D2 blockade delayed the decision to start and stop responding in both regions, and this effect was larger in the dorsomedial striatum. By contrast, dorsomedial D1 blockade delayed stop times, without significantly delaying start times, whereas dorsolateral D1 blockade produced no detectable effects. These findings suggest that striatal dopamine may tune decision thresholds during timing tasks. Furthermore, our data indicate that the dorsomedial striatum plays a key role in temporal control, which may be useful for localizing neural circuits that mediate the temporal control of action.

Lesions of ventrolateral striatum eliminate lose-shift but not win-stay behaviour in rats

Animals tend to repeat actions that are associated with reward delivery, whereas they tend to shift responses to alternate choices following reward omission. These so-called win-stay and lose-shift responses are employed by a wide range of animals in a variety of decision-making scenarios, and depend on dissociated regions of the striatum. Specifically, lose-shift responding is impaired by extensive excitotoxic lesions of the lateral striatum. Here we used focal lesions to assess whether dorsal and ventral regions of the lateral striatum contribute differently to this effect. We found that damage to ventrolateral striatum reduced lose-shift responding without impairing win-stay, motoric, or motivational aspects of behaviour in the task, whereas lesions confined to the dorsolateral striatum significantly impaired the ability of rats to complete trials of the task. Moreover, lesions to the dorsomedial striatum had no effect on either lose-shift or win-stay responding. Together, these data suggest a novel role of the ventral portion of the lateral striatum in driving lose-shift decisions.

The impact of drugs of abuse on executive function: characterizing long-term changes in neural correlates following chronic drug exposure and withdrawal in rats

Addiction has long been characterized by diminished executive function, control, and impulsivity management. In particular, these deficits often manifest themselves as impairments in reversal learning, delay discounting, and response inhibition. Understanding the neurobiological substrates of these behavioral deficits is of paramount importance to our understanding of addiction. Within the cycle of addiction, periods during and after withdrawal represent a particularly difficult point of intervention in that the negative physical symptoms associated with drug removal and drug craving increase the likelihood that the patient will relapse and return to drug use in order to abate these symptoms. Moreover, it is often during this time that drug induced deficits in executive function hinder the ability of the patient to refrain from drug use. Thus, it is necessary to understand the physiological and behavioral changes associated with withdrawal and drug craving-largely manifesting as deficits in executive control-to develop more effective treatment strategies. In this review, we address the long-term impact that drugs of abuse have on the behavioral and neural correlates that give rise to executive control as measured by reversal learning, delay discounting, and stop-signal tasks, focusing particularly on our work using rats as a model system.

Striatal action-value neurons reconsidered

It is generally believed that during economic decisions, striatal neurons represent the values associated with different actions. This hypothesis is based on studies, in which the activity of striatal neurons was measured while the subject was learning to prefer the more rewarding action. Here we show that these publications are subject to at least one of two critical confounds. First, we show that even weak temporal correlations in the neuronal data may result in an erroneous identification of action-value representations. Second, we show that experiments and analyses designed to dissociate action-value representation from the representation of other decision variables cannot do so. We suggest solutions to identifying action-value representation that are not subject to these confounds. Applying one solution to previously identified action-value neurons in the basal ganglia we fail to detect action-value representations. We conclude that the claim that striatal neurons encode action-values must await new experiments and analyses.

Dorsolateral Striatum Engagement Interferes with Early Discrimination Learning

In current models, learning the relationship between environmental stimuli and the outcomes of actions involves both stimulus-driven and goal-directed systems, mediated in part by the DLS and DMS, respectively. However, though these models emphasize the importance of the DLS in governing actions after extensive experience has accumulated, there is growing evidence of DLS engagement from the onset of training. Here, we used in vivo photosilencing to reveal that DLS recruitment interferes with early touchscreen discrimination learning. We also show that the direct output pathway of the DLS is preferentially recruited and causally involved in early learning and find that silencing the normal contribution of the DLS produces plasticity-related alterations in a PL-DMS circuit. These data provide further evidence suggesting that the DLS is recruited in the construction of stimulus-elicited actions that ultimately automate behavior and liberate cognitive resources for other demands, but with a cost to performance at the outset of learning.

Orbitofrontal neurons signal reward predictions, not reward prediction errors

Neurons in the orbitofrontal cortex (OFC) fire in anticipation of and during rewards. Such firing has been suggested to encode reward predictions and to account in some way for the role of this area in adaptive behavior and learning. However, it has also been reported that neural activity in OFC reflects reward prediction errors, which might drive learning directly. Here we tested this question by analyzing the firing of OFC neurons recorded in an odor discrimination task in which rats were trained to sample odor cues and respond left or right on each trial for reward. Neurons were recorded across blocks of trials in which we switched either the number or the flavor of the reward delivered in each well. Previously we have described how neurons in this dataset fired to the predictive cues (Stalnaker et al., 2014); here we focused on the firing in anticipation of and just after delivery of each drop of reward, looking specifically for differences in firing based on whether the reward number or flavor was unexpected or expected. Unlike dopamine neurons recorded in this setting, which exhibited phasic error-like responses after surprising changes in either reward number or reward flavor (Takahashi et al., 2017), OFC neurons showed no such error correlates and instead fired in a way that reflected reward predictions.

Avolition-Apathy and White Matter Connectivity in Schizophrenia: Reduced Fractional Anisotropy Between Amygdala and Insular Cortex

The avolition/apathy domain of negative symptoms includes motivation- and pleasure-related impairments. In people with schizophrenia, structural and functional abnormalities were reported in key regions within the motivational reward system, including ventral-tegmental area (VTA), striatum (especially at the level of the nucleus accumbens, NAcc), orbitofrontal cortex (OFC), as well as amygdala (Amy) and insular cortex (IC). However, the association of the reported abnormalities with avoliton-apathy is still controversial. In the present study, we investigated white matter connectivity patterns within these regions, using a probabilistic analysis of diffusion tensor imaging (DTI) data, in male subjects with schizophrenia. Thirty-five male subjects with schizophrenia (SCZ) and 17 male healthy controls (HC) matched for age, underwent DTI. SCZ were evaluated using the Schedule for Deficit Syndrome (SDS), the Positive and Negative Syndrome Scale (PANSS), and the MATRICS Consensus Cognitive Battery (MCCB). Probabilistic tractography was applied to investigate pathways connecting the Amy and the NAcc with the OFC and IC. Reduced fractional anisotropy (FA) was observed in left Amy-ventral anterior IC connections, in SCZ compared with controls. This abnormality was negatively correlated with avolition/apathy but not with expressive deficit scores. SCZ showed also a reduced connectivity index between right NAcc and medial OFC, as compared with controls. Finally, the left NAcc-dorsal anterior IC connectivity index was negatively correlated with working memory scores. Our results indicate that only the avolition/apathy domain of negative symptoms is related to abnormal connectivity in the motivation-related circuits. The findings also demonstrate that distinct alterations underlie cognitive impairment and avolition/apathy.

Inter-individual differences in decision-making, flexible and goal-directed behaviors: novel insights within the prefronto-striatal networks

Inflexible behavior is a hallmark of several decision-making-related disorders such as ADHD and addiction. As in humans, a subset of healthy rats makes poor decisions and prefers immediate larger rewards despite suffering large losses in a rat gambling task (RGT). They also display a combination of traits reminiscent of addiction, notably inflexible behavior and perseverative responses. The goal of the present work was twofold: (1) to elucidate if behavioral inflexibility of poor decision-makers could be related to a lower quality of goal-directed behavior (action-outcome associations); (2) to uncover the neural basis of inter-individual differences in goal-directed behavior. We specifically assessed inter-individual differences in decision-making in the RGT, flexibility in the RGT-reversed version and goal-directed behavior in a contingency degradation test, i.e., response adaptation when dissociating reward delivery from the animal's action. The contributions of the medial prefrontal cortex and the dorsal striatum to action-outcome associations were assessed using Zif268 immunodetection. Inflexible behavior was related to a lower sensitivity to contingency degradation in all poor decision-makers and only in a few good decision-makers. This poorer sensitivity was associated with a lower immunoreactivity in prelimbic and infralimbic cortices and a higher one in the dorsomedial and dorsolateral striatum. These findings suggest that an imbalanced prefronto-striatal activity could underlie inaccurate goal representation in changing environments and may promote maladaptive habit formation among poor decision-makers. These data strengthen our previous work identifying biomarkers of vulnerability to develop psychiatric disorders and demonstrate the relevance of inter-individual differences to model maladaptive behaviors.

Changes in striatal activity and functional connectivity in a mouse model of Huntington's disease

Hereditary Huntington’s disease (HD) is associated with progressive motor, cognitive and psychiatric symptoms. A primary consequence of the HD mutation is the preferential loss of medium spiny projection cells with relative sparing of local interneurons in the striatum. In addition, among GABAergic striatal projection cells, indirect pathway cells expressing D2 dopamine receptors are lost earlier than direct pathway cells expressing D1 receptors. To test in vivo the functional integrity of direct and indirect pathways as well as interneurons in the striatum of male R6/1 transgenic mice, we assessed their c-Fos expression levels induced by a striatal-dependent cognitive task and compared them with age-matched wild-type littermates. We found a significant increase of c-Fos+ nuclei in the dorsomedial striatum, and this only at 2 months, when our HD mouse model is still pre-motor symptomatic, the increase disappearing with symptom manifestation. Contrary to our expectation, the indirect pathway projection neurons did not undergo any severer changes of c-Fos expression regardless of age in R6/1 mice. We also found a decreased activation of interneurons that express parvalbumin in the dorsomedial striatum at both presymptomatic and symptomatic ages. Finally, analysis of c-Fos expression in extended brain regions involved in the cognitive learning used in our study, demonstrates, throughout ages studied, changes in the functional connectivity between regions in the transgenic mice. Further analysis of the cellular and molecular changes underlying the transient striatal hyperactivity in the HD mice may help to understand the mechanisms involved in the disease onset.

Striatal GPR88 Modulates Foraging Efficiency

The striatum is anatomically and behaviorally implicated in behaviors that promote efficient foraging. To investigate this function, we studied instrumental choice behavior in mice lacking GPR88, a striatum-enriched orphan G-protein-coupled receptor that modulates striatal medium spiny neuron excitability. Our results reveal that hungry mice lacking GPR88 (KO mice) were slow to acquire food-reinforced lever press but could lever press similar to controls on a progressive ratio schedule. Both WT and KO mice discriminated between reward and no-reward levers; however, KO mice failed to discriminate based on relative quantity-reward (1 vs 3 food pellets) or effort (3 vs 9 lever presses). We also demonstrate preference for the high-reward (3 pellet) lever was selectively reestablished when GPR88 expression was restored to the striatum. We propose that GPR88 expression within the striatum is integral to efficient action-selection during foraging.SIGNIFICANCE STATEMENT Evolutionary pressure driving energy homeostasis favored detection and comparison of caloric value. In wild and laboratory settings, neural systems involved in energy homeostasis bias foraging to maximize energy efficiency. This is observed when foraging behaviors are guided by superior nutritional density or minimized caloric expenditure. The striatum is anatomically and functionally well placed to perform the sensory and motor integration necessary for efficient action selection during foraging. However, few studies have examined this behavioral phenomenon or elucidated underlying molecular mechanisms. Both humans and mice with nonfunctional GPR88 have been shown to present striatal dysfunctions and impaired learning. We demonstrate that GPR88 expression is necessary to efficiently integrate effort and energy density information guiding instrumental choice.

Prior Cocaine Self-Administration Increases Response-Outcome Encoding That Is Divorced from Actions Selected in Dorsal Lateral Striatum

Dorsal lateral striatum (DLS) is a highly associative structure that encodes relationships among environmental stimuli, behavioral responses, and predicted outcomes. DLS is known to be disrupted after chronic drug abuse; however, it remains unclear what neural signals in DLS are altered. Current theory suggests that drug use enhances stimulus-response processing at the expense of response-outcome encoding, but this has mostly been tested in simple behavioral tasks. Here, we investigated what neural correlates in DLS are affected by previous cocaine exposure as rats performed a complex reward-guided decision-making task in which predicted reward value was independently manipulated by changing the delay to or size of reward associated with a response direction across a series of trial blocks. After cocaine self-administration, rats exhibited stronger biases toward higher-value reward and firing in DLS more strongly represented action-outcome contingencies independent from actions subsequently taken rather than outcomes predicted by selected actions (chosen-outcome contingencies) and associations between stimuli and actions (stimulus-response contingencies). These results suggest that cocaine self-administration strengthens action-outcome encoding in rats (as opposed to chosen-outcome or stimulus-response encoding), which abnormally biases behavior toward valued reward when there is a choice between two options during reward-guided decision-making.SIGNIFICANCE STATEMENT Current theories suggest that the impaired decision-making observed in individuals who chronically abuse drugs reflects a decrease in goal-directed behaviors and an increase in habitual behaviors governed by neural representations of response-outcome (R-O) and stimulus-response associations, respectively. We examined the impact that prior cocaine self-administration had on firing in dorsal lateral striatum (DLS), a brain area known to be involved in habit formation and affected by drugs of abuse, during performance of a complex reward-guided decision-making task. Surprisingly, we found that previous cocaine exposure enhanced R-O associations in DLS. This suggests that there may be more complex consequences of drug abuse than current theories have explored, especially when examining brain and behavior in the context of a complex two-choice decision-making task.

Cholinergic Interneurons Use Orbitofrontal Input to Track Beliefs about Current State

When conditions change, organisms need to learn about the changed conditions without interfering with what they already know. To do so, they can assign the new learning to a new "state" and the old learning to a previous state. This state assignment is fundamental to behavioral flexibility. Cholinergic interneurons (CINs) in the dorsomedial striatum (DMS) are necessary for associative information to be compartmentalized in this way, but the mechanism by which they do so is unknown. Here we addressed this question by recording putative CINs from the DMS in rats performing a task consisting of a series of trial blocks, or states, that required the recall and application of contradictory associative information. We found that individual CINs in the DMS represented the current state throughout each trial. These state correlates were not observed in dorsolateral striatal CINs recorded in the same rats. Notably, DMS CIN ensembles tracked rats' beliefs about the current state such that, when states were miscoded, rats tended to make suboptimal choices reflecting the miscoding. State information held by the DMS CINs also depended completely on the orbitofrontal cortex, an area that has been proposed to signal environmental states. These results suggest that CINs set the stage for recalling associative information relevant to the current environment by maintaining a real-time representation of the current state. Such a role has novel implications for understanding the neural basis of a variety of psychiatric diseases, such as addiction or anxiety disorders, in which patients generalize inappropriately (or fail to generalize) between different environments.

SIGNIFICANCE STATEMENT

Striatal cholinergic interneurons (CINs) are thought to be identical to tonically active neurons. These neurons have long been thought to have an important influence on striatal processing during reward-related learning. Recently, a more specific function for striatal CINs has been suggested, which is that they are necessary for striatal learning to be compartmentalized into different states as the state of the environment changes. Here we report that putative CINs appear to track rats' beliefs about which environmental state is current. We further show that this property of CINs depends on orbitofrontal cortex input and is correlated with choices made by rats. These findings could provide new insight into neuropsychiatric diseases that involve improper generalization between different contexts.

A neuronal activation correlate in striatum and prefrontal cortex of prolonged cocaine intake

Maladaptive changes in the involvement of striatal and frontal cortical regions in drug use are thought to underlie the progression to habitual drug use and loss of cognitive control over drug intake that occur with accumulating drug experience. The present experiments focus on changes in neuronal activity in these regions associated with short-term (10 days) and long-term (60 days) self-administration of cocaine. Quantitative in situ hybridization for the immediate early gene Mkp1 was combined with statistical parametric mapping to assess the distribution of neuronal activity. We hypothesized that neuronal activity in striatum would increase in its dorsal part and that activity in frontal cortex would decrease with prolonged cocaine self-administration experience. Expression of Mkp1 was profoundly increased after cocaine self-administration, and the magnitude of this effect was greater after short-term compared to long-term self-administration. Increased neuronal activity was seen in both dorsal and ventral sectors of the striatum after 10 days exposure to cocaine. However, enhanced activity was restricted to dorsomedial and dorsocentral striatum after 60 days cocaine self-administration. In virtually all medial prefrontal and most orbitofrontal areas, increased expression of Mkp1 was observed after 10 days of cocaine taking, whereas after 60 days, enhanced expression was restricted to caudal parts of medial prefrontal and caudomedial parts of orbitofrontal cortex. Our data reveal functional changes in cellular activity in striatum and frontal cortex with increasing cocaine self-administration experience. These changes might reflect the neural processes that underlie the descent from recreational drug taking to compulsive cocaine use.

Habit formation

Habits, both good ones and bad ones, are pervasive in animal behavior. Important frameworks have been developed to understand habits through psychological and neurobiological studies. This work has given us a rich understanding of brain networks that promote habits, and has also helped us to understand what constitutes a habitual behavior as opposed to a behavior that is more flexible and prospective. Mounting evidence from studies using neural recording methods suggests that habit formation is not a simple process. We review this evidence and take the position that habits could be sculpted from multiple dissociable changes in neural activity. These changes occur across multiple brain regions and even within single brain regions. This strategy of classifying components of a habit based on different brain signals provides a potentially useful new way to conceive of disorders that involve overly fixed behaviors as arising from different potential dysfunctions within the brain's habit network.

Dissociated sequential activity and stimulus encoding in the dorsomedial striatum during spatial working memory

Several lines of evidence suggest that the striatum has an important role in spatial working memory. The neural dynamics in the striatum have been described in tasks with short delay periods (1–4 s), but remain largely uncharacterized for tasks with longer delay periods. We collected and analyzed single unit recordings from the dorsomedial striatum of rats performing a spatial working memory task with delays up to 10 s. We found that neurons were activated sequentially, with the sequences spanning the entire delay period. Surprisingly, this sequential activity was dissociated from stimulus encoding activity, which was present in the same neurons, but preferentially appeared towards the onset of the delay period. These observations contrast with descriptions of sequential dynamics during similar tasks in other brains areas, and clarify the contribution of the striatum to spatial working memory.

DOI:http://dx.doi.org/10.7554/eLife.19507.001

Associative and sensorimotor cortico-basal ganglia circuit roles in effects of abused drugs

The mammalian forebrain is characterized by the presence of several parallel cortico-basal ganglia circuits that shape the learning and control of actions. Among these are the associative, limbic and sensorimotor circuits. The function of all of these circuits has now been implicated in responses to drugs of abuse, as well as drug seeking and drug taking. While the limbic circuit has been most widely examined, key roles for the other two circuits in control of goal-directed and habitual instrumental actions related to drugs of abuse have been shown. In this review we describe the three circuits and effects of acute and chronic drug exposure on circuit physiology. Our main emphasis is on drug actions in dorsal striatal components of the associative and sensorimotor circuits. We then review key findings that have implicated these circuits in drug seeking and taking behaviors, as well as drug use disorders. Finally, we consider different models describing how the three cortico-basal ganglia circuits become involved in drug-related behaviors. This topic has implications for drug use disorders and addiction, as treatments that target the balance between the different circuits may be useful for reducing excessive substance use.

Neuronal activity in dorsomedial and dorsolateral striatum under the requirement for temporal credit assignment

To investigate neural processes underlying temporal credit assignment in the striatum, we recorded neuronal activity in the dorsomedial and dorsolateral striatum (DMS and DLS, respectively) of rats performing a dynamic foraging task in which a choice has to be remembered until its outcome is revealed for correct credit assignment. Choice signals appeared sequentially, initially in the DMS and then in the DLS, and they were combined with action value and reward signals in the DLS when choice outcome was revealed. Unlike in conventional dynamic foraging tasks, neural signals for chosen value were elevated in neither brain structure. These results suggest that dynamics of striatal neural signals related to evaluating choice outcome might differ drastically depending on the requirement for temporal credit assignment. In a behavioral context requiring temporal credit assignment, the DLS, but not the DMS, might be in charge of updating the value of chosen action by integrating choice, action value, and reward signals together.

Endocannabinoid Modulation of Orbitostriatal Circuits Gates Habit Formation

Everyday function demands efficient and flexible decision-making that allows for habitual and goal-directed action control. An inability to shift has been implicated in disorders with impaired decision-making, including obsessive-compulsive disorder and addiction. Despite this, our understanding of the specific molecular mechanisms and circuitry involved in shifting action control remains limited. Here we identify an endogenous molecular mechanism in a specific cortical-striatal pathway that mediates the transition between goal-directed and habitual action strategies. Deletion of cannabinoid type 1 (CB1) receptors from cortical projections originating in the orbital frontal cortex (OFC) prevents mice from shifting from goal-directed to habitual instrumental lever pressing. Activity of OFC neurons projecting to dorsal striatum (OFC-DS) and, specifically, activity of OFC-DS terminals is necessary for goal-directed action control. Lastly, CB1 deletion from OFC-DS neurons prevents the shift from goal-directed to habitual action control. These data suggest that the emergence of habits depends on endocannabinoid-mediated attenuation of a competing circuit controlling goal-directed behaviors.

Dopamine Prediction Errors in Reward Learning and Addiction: From Theory to Neural Circuitry

Midbrain dopamine (DA) neurons are proposed to signal reward prediction error (RPE), a fundamental parameter in associative learning models. This RPE hypothesis provides a compelling theoretical framework for understanding DA function in reward learning and addiction. New studies support a causal role for DA-mediated RPE activity in promoting learning about natural reward; however, this question has not been explicitly tested in the context of drug addiction. In this review, we integrate theoretical models with experimental findings on the activity of DA systems, and on the causal role of specific neuronal projections and cell types, to provide a circuit-based framework for probing DA-RPE function in addiction. By examining error-encoding DA neurons in the neural network in which they are embedded, hypotheses regarding circuit-level adaptations that possibly contribute to pathological error signaling and addiction can be formulated and tested.

Translational Research on Habit and Alcohol

Habitual actions enable efficient daily living, but they can also contribute to pathological behaviors that resistant change, such as alcoholism. Habitual behaviors are learned actions that appear goal-directed but are in fact no longer under the control of the action's outcome. Instead, these actions are triggered by stimuli, which may be exogenous or interoceptive, discrete or contextual. A major hallmark characteristic of alcoholism is continued alcohol use despite serious negative consequences. In essence, although the outcome of alcohol seeking and drinking is dramatically devalued, these actions persist, often triggered by environmental cues associated with alcohol use. Thus, alcoholism meets the definition of an initially goal-directed behavior that converts to a habit-based process. Habit and alcohol have been well investigated in rodent models, with comparatively less research in non-human primates and people. This review focuses on translational research on habit and alcohol with an emphasis on cross-species methodology and neural circuitry.

Distinct neural representation in the dorsolateral, dorsomedial, and ventral parts of the striatum during fixed- and free-choice tasks

The striatum is a major input site of the basal ganglia, which play an essential role in decision making. Previous studies have suggested that subareas of the striatum have distinct roles: the dorsolateral striatum (DLS) functions in habitual action, the dorsomedial striatum (DMS) in goal-directed actions, and the ventral striatum (VS) in motivation. To elucidate distinctive functions of subregions of the striatum in decision making, we systematically investigated information represented by phasically active neurons in DLS, DMS, and VS. Rats performed two types of choice tasks: fixed- and free-choice tasks. In both tasks, rats were required to perform nose poking to either the left or right hole after cue-tone presentation. A food pellet was delivered probabilistically depending on the presented cue and the selected action. The reward probability was fixed in fixed-choice task and varied in a block-wise manner in free-choice task. We found the following: (1) when rats began the tasks, a majority of VS neurons increased their firing rates and information regarding task type and state value was most strongly represented in VS; (2) during action selection, information of action and action values was most strongly represented in DMS; (3) action-command information (action representation before action selection) was stronger in the fixed-choice task than in the free-choice task in both DLS and DMS; and (4) action-command information was strongest in DLS, particularly when the same choice was repeated. We propose a hypothesis of hierarchical reinforcement learning in the basal ganglia to coherently explain these results.