Signals for previous goal choice persist in the dorsomedial, but not dorsolateral striatum of rats

Chronic Ethanol Exposure Produces Persistent Impairment in Cognitive Flexibility and Decision Signals in the Striatum

Lack of cognitive flexibility is a hallmark of substance use disorders and has been associated with drug-induced synaptic plasticity in the dorsomedial striatum (DMS). Yet the possible impact of altered plasticity on real-time striatal neural dynamics during decision-making is unclear. Here, we identified persistent impairments induced by chronic ethanol (EtOH) exposure on cognitive flexibility and striatal decision signals. After a substantial withdrawal period from prior EtOH vapor exposure, male, but not female, rats exhibited reduced adaptability and exploratory behavior during a dynamic decision-making task. Reinforcement learning models showed that prior EtOH exposure enhanced learning from rewards over omissions. Notably, neural signals in the DMS related to the decision outcome were enhanced, while those related to choice and choice-outcome conjunction were reduced, in EtOH-treated rats compared to the controls. These findings highlight the profound impact of chronic EtOH exposure on adaptive decision-making, pinpointing specific changes in striatal representations of actions and outcomes as underlying mechanisms for cognitive deficits.

Neural circuit models for evidence accumulation through choice-selective sequences

Decision making is traditionally thought to be mediated by populations of neurons whose firing rates persistently accumulate evidence across time. However, recent decision-making experiments in rodents have observed neurons across the brain that fire sequentially as a function of spatial position or time, rather than persistently, with the subset of neurons in the sequence depending on the animal's choice. We develop two new candidate circuit models, in which evidence is encoded either in the relative firing rates of two competing chains of neurons or in the network location of a stereotyped pattern ("bump") of neural activity. Encoded evidence is then faithfully transferred between neuronal populations representing different positions or times. Neural recordings from four different brain regions during a decision-making task showed that, during the evidence accumulation period, different brain regions displayed tuning curves consistent with different candidate models for evidence accumulation. This work provides mechanistic models and potential neural substrates for how graded-value information may be precisely accumulated within and transferred between neural populations, a set of computations fundamental to many cognitive operations.

Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.

The learning of prospective and retrospective cognitive maps within neural circuits

Brain circuits are thought to form a "cognitive map" to process and store statistical relationships in the environment. A cognitive map is commonly defined as a mental representation that describes environmental states (i.e., variables or events) and the relationship between these states. This process is commonly conceptualized as a prospective process, as it is based on the relationships between states in chronological order (e.g., does reward follow a given state?). In this perspective, we expand this concept on the basis of recent findings to postulate that in addition to a prospective map, the brain forms and uses a retrospective cognitive map (e.g., does a given state precede reward?). In doing so, we demonstrate that many neural signals and behaviors (e.g., habits) that seem inflexible and non-cognitive can result from retrospective cognitive maps. Together, we present a significant conceptual reframing of the neurobiological study of associative learning, memory, and decision making.

Dorsolateral Striatal Task-initiation Bursts Represent Past Experiences More than Future Action Plans

The dorsolateral striatum (DLS) is involved in learning and executing procedural actions. Cell ensembles in the DLS, but not the dorsomedial striatum (DMS), exhibit a burst of firing at the start of a well-learned action sequence ("task-bracketing"). However, it is currently unclear what information is contained in these bursts. Some theories suggest that these bursts should represent the procedural action sequence itself (that they should be about future action chains), whereas others suggest that they should contain representations of the current state of the world, taking into account primarily past information. In addition, the DLS local field potential shows transient bursts of power in the 50 Hz range (γ50) around the time a learned action sequence is initiated. However, it is currently unknown how bursts of activity in DLS cell ensembles and bursts of γ50 power in the DLS local field potential are related to each other. We found that DLS bursts at lap initiation in rats represented recently experienced reward locations more than future procedural actions, indicating that task-initiation DLS bursts contain primarily retrospective, rather than prospective, information to guide procedural actions. Furthermore, representations of past reward locations increased during periods of increased γ50 power in the DLS. There was no evidence of task-initiation bursts, increased γ50 power, or retrospective reward location information in the neighboring dorsomedial striatum. These data support a role for the DLS in model-free theories of procedural decision-making over planned action-chain theories, suggesting that procedural actions derive from representations of the current and recent past.SIGNIFICANCE STATEMENT While it is well-established that the dorsolateral striatum (DLS) plays a critical role in procedural decision-making, open questions remain about the kinds of representations contained in DLS ensemble activity that guide procedural actions. We found that DLS, but not DMS, cell ensembles contained nonlocal representations of past reward locations that appear moments before task-initiation DLS bursts. These retrospective representations were temporally linked to a rise in γ50 power that also preceded the characteristic DLS burst at task-initiation. These results support models of procedural decision-making based on associations between available actions and the current state of the world over models based on planning over action-chains.

Depression in the Direct Pathway of the Dorsomedial Striatum Permits the Formation of Habitual Action

In order to achieve optimal outcomes in an ever-changing environment, humans and animals generally manage their action control via either goal-directed action or habitual action. These two action strategies are thought to be encoded in distinct parallel circuits in the dorsal striatum, specifically, the posterior dorsomedial striatum (DMS) and the dorsolateral striatum (DLS), respectively. The striatum is primarily composed of two subtypes of medium spiny neurons (MSNs): the direct-pathway striatonigral and the indirect-pathway striatopallidal MSNs. MSN-subtype-specific synaptic plasticity in the DMS and the DLS has been revealed to underlie goal-directed action and habitual action, respectively. However, whether any MSN-subtype-specific synaptic plasticity in the DMS is associated with habitual action, and if so, whether the synaptic plasticity affects the formation of habitual action, are not known. This study demonstrates that postsynaptic depression in the excitatory synapses of the direct-pathway striatonigral MSNs in the DMS is formed after habit learning. Moreover, chemogenetically rescuing this depression compromises the acquisition, but not the expression, of habitual action. These findings reveal that an MSN-subtype-specific synaptic plasticity in the DMS affects habitual action and suggest that plasticity in the DMS as well as in the DLS contributes to the formation of habitual action.

Robust and distributed neural representation of action values

Studies in rats, monkeys, and humans have found action-value signals in multiple regions of the brain. These findings suggest that action-value signals encoded in these brain structures bias choices toward higher expected rewards. However, previous estimates of action-value signals might have been inflated by serial correlations in neural activity and also by activity related to other decision variables. Here, we applied several statistical tests based on permutation and surrogate data to analyze neural activity recorded from the striatum, frontal cortex, and hippocampus. The results show that previously identified action-value signals in these brain areas cannot be entirely accounted for by concurrent serial correlations in neural activity and action value. We also found that neural activity related to action value is intermixed with signals related to other decision variables. Our findings provide strong evidence for broadly distributed neural signals related to action value throughout the brain.

Dynamic decision making and value computations in medial frontal cortex

Dynamic decision making requires an intact medial frontal cortex. Recent work has combined theory and single-neuron measurements in frontal cortex to advance models of decision making. We review behavioral tasks that have been used to study dynamic decision making and algorithmic models of these tasks using reinforcement learning theory. We discuss studies linking neurophysiology and quantitative decision variables. We conclude with hypotheses about the role of other cortical and subcortical structures in dynamic decision making, including ascending neuromodulatory systems.

Active maintenance of eligibility trace in rodent prefrontal cortex

Even though persistent neural activity has been proposed as a mechanism for maintaining eligibility trace, direct empirical evidence for active maintenance of eligibility trace has been lacking. We recorded neuronal activity in the medial prefrontal cortex (mPFC) in rats performing a dynamic foraging task in which a choice must be remembered until its outcome on the timescale of seconds for correct credit assignment. We found that mPFC neurons maintain significant choice signals during the time period between action selection and choice outcome. We also found that neural signals for choice, outcome, and action value converge in the mPFC when choice outcome was revealed. Our results indicate that the mPFC maintains choice signals necessary for temporal credit assignment in the form of persistent neural activity in our task. They also suggest that the mPFC might update action value by combining actively maintained eligibility trace with action value and outcome signals.

Complex Movement Control in a Rat Model of Parkinsonian Falls: Bidirectional Control by Striatal Cholinergic Interneurons

Older persons and, more severely, persons with Parkinson's disease (PD) exhibit gait dysfunction, postural instability and a propensity for falls. These dopamine (DA) replacement-resistant symptoms are associated with losses of basal forebrain and striatal cholinergic neurons, suggesting that falls reflect disruption of the corticostriatal transfer of movement-related cues and their striatal integration with movement sequencing. To advance a rodent model of the complex movement deficits of Parkinsonian fallers, here we first demonstrated that male and female rats with dual cortical cholinergic and striatal DA losses (DL rats) exhibit cued turning deficits, modeling the turning deficits seen in these patients. As striatal cholinergic interneurons (ChIs) are positioned to integrate movement cues with gait, and as ChI loss has been associated with falls in PD, we next used this task, as well as a previously established task used to reveal heightened fall rates in DL rats, to broadly test the role of ChIs. Chemogenetic inhibition of ChIs in otherwise intact male and female rats caused cued turning deficits and elevated fall rates. Spontaneous turning was unaffected. Furthermore, chemogenetic stimulation of ChIs in DL rats reduced fall rates and restored cued turning performance. Stimulation of ChIs was relatively more effective in rats with viral transfection spaces situated lateral to the DA depletion areas in the dorsomedial striatum. These results indicate that striatal ChIs are essential for the control of complex movements, and they suggest a therapeutic potential of stimulation of ChIs to restore gait and balance, and to prevent falls in PD.SIGNIFICANCE STATEMENT In persons with Parkinson's disease, gait dysfunction and the associated risk for falls do not benefit from dopamine replacement therapy and often result in long-term hospitalization and nursing home placement. Here, we first validated a new task to demonstrate impairments in cued turning behavior in rodents modeling the cholinergic-dopaminergic losses observed in Parkinsonian fallers. We then demonstrated the essential role of striatal cholinergic interneurons for turning behavior as well as for traversing dynamic surfaces and avoiding falls. Stimulation of these interneurons in the rat model rescued turning performance and reduced fall rates. Our findings indicate the feasibility of investigating the neuronal circuitry underling complex movement control in rodents, and that striatal cholinergic interneurons are an essential node of such circuitry.

Temporal dynamics of Arc/Arg3.1 expression in the dorsal striatum during acquisition and consolidation of a motor skill in mice

Region- and pathway-specific plasticity within striatal circuits is critically involved in the acquisition and long-term retention of a new motor skill as it becomes automatized. However, the molecular substrates contributing to this plasticity remain unclear. Here, we examined the expression of the activity-regulated cytoskeleton-associated protein (Arc) in the associative or dorsomedial striatum (DMS) and the sensorimotor or dorsolateral striatum (DLS), as well as in striatonigral and striatopallidal neurons, during different skill learning phases in the accelerating rotarod task. We found that Arc was mainly expressed in the DMS during early motor learning and progressively increased in the DLS during gradual motor skill consolidation. Moreover, Arc was preferentially expressed in striatopallidal neurons early in training and gradually increased in striatonigral neurons later in training. These data demonstrate that in the dorsal striatum, the expression of Arc exhibits a region- and cell-specific transfer during the learning of a motor skill, suggesting a link between striatal Arc expression and motor skill learning in mice.

No substantial change in the balance between model-free and model-based control via training on the two-step task

Human decisions can be habitual or goal-directed, also known as model-free (MF) or model-based (MB) control. Previous work suggests that the balance between the two decision systems is impaired in psychiatric disorders such as compulsion and addiction, via overreliance on MF control. However, little is known whether the balance can be altered through task training. Here, 20 healthy participants performed a well-established two-step task that differentiates MB from MF control, across five training sessions. We used computational modelling and functional near-infrared spectroscopy to assess changes in decision-making and brain hemodynamic over time. Mixed-effects modelling revealed overall no substantial changes in MF and MB behavior across training. Although our behavioral and brain findings show task-induced changes in learning rates, these parameters have no direct relation to either MF or MB control or the balance between the two systems, and thus do not support the assumption of training effects on MF or MB strategies. Our findings indicate that training on the two-step paradigm in its current form does not support a shift in the balance between MF and MB control. We discuss these results with respect to implications for restoring the balance between MF and MB control in psychiatric conditions.

Psychiatric conditions such as compulsion or addiction are associated with an overreliance on habitual, or model-free, decision-making. Goal-directed, or model-based, decision-making may protect against such overreliance. We therefore asked whether model-free control could be reduced, and model-based control strengthened, via task training. We used the well-characterized two-step task that differentiates model-based from model-free actions. Our results suggest that training on the current form of the two-step task does not support a shift in the balance between model-free and model-based strategies. Factors such as devaluation, demotivation or automatization during training may play a role in the missing emergence of a training effect. Future studies could adapt the two-step task so as to separate such factors from decision-making strategies.

Enhanced Population Coding for Rewarded Choices in the Medial Frontal Cortex of the Mouse

Instrumental behavior is characterized by the selection of actions based on the degree to which they lead to a desired outcome. However, we lack a detailed understanding of how rewarded actions are reinforced and preferentially implemented. In rodents, the medial frontal cortex is hypothesized to play an important role in this process, based in part on its capacity to encode chosen actions and their outcomes. We therefore asked how neural representations of choice and outcome might interact to facilitate instrumental behavior. To investigate this question, we imaged neural ensemble activity in layer 2/3 of the secondary motor region (M2) while mice engaged in a two-choice auditory discrimination task with probabilistic outcomes. Correct choices could result in one of three reward amounts (single, double or omitted reward), which allowed us to measure neural and behavioral effects of reward magnitude, as well as its categorical presence or absence. Single-unit and population decoding analyses revealed a consistent influence of outcome on choice signals in M2. Specifically, rewarded choices were more robustly encoded relative to unrewarded choices, with little dependence on the exact magnitude of reinforcement. Our results provide insight into the integration of past choices and outcomes in the rodent brain during instrumental behavior.

Distributed Encoding of Reinforcement in Rat Cortico-Striatal-Limbic Networks

Decision-making in the mammalian brain typically involves multiple brain structures within the midbrain, thalamus, striatum, limbic system, and cortex. Although task specific contributions of each brain region have been identified, neurons responding to reinforcement have been found throughout these structures. We sought to determine if any brain area, or cluster of areas, are the source of information, and if the fidelity of information varies among the areas. We recorded simultaneous field potentials (FPs) in rats from seven brain regions as they completed a binary choice task. The FPs of a 0.5 s window following reinforcement were given as input to a classifier that attempted to predict whether or not the rat received reward on each trial. The classifier correctly categorized reward on 77% of trials. Any region-specific signal could be omitted without lowering accuracy. Frequencies above 40 Hz and signals recorded later than 0.25 s following reinforcement were necessary to achieve this accuracy. Further, the classifier was able to predict reinforcement outcome above chance levels when using FPs from any single recorded brain region. Some combinations of structures, however, were more predictive than others. Analysis of FPs prior to reward revealed most regions reflected the prior probability of reward. Lastly, analyses of information flow suggested reinforcement information does not originate within a single structure of the network, within the resolution afforded by FP recordings. These data suggest reward delivery information is rapidly distributed non-uniformly across the network, and there is no canonical flow of information about reward events in the recorded structures.

Stable Representations of Decision Variables for Flexible Behavior

Decisions occur in dynamic environments. In the framework of reinforcement learning, the probability of performing an action is influenced by decision variables. Discrepancies between predicted and obtained rewards (reward prediction errors) update these variables, but they are otherwise stable between decisions. Although reward prediction errors have been mapped to midbrain dopamine neurons, it is unclear how the brain represents decision variables themselves. We trained mice on a dynamic foraging task in which they chose between alternatives that delivered reward with changing probabilities. Neurons in the medial prefrontal cortex, including projections to the dorsomedial striatum, maintained persistent firing rate changes over long timescales. These changes stably represented relative action values (to bias choices) and total action values (to bias response times) with slow decay. In contrast, decision variables were weakly represented in the anterolateral motor cortex, a region necessary for generating choices. Thus, we define a stable neural mechanism to drive flexible behavior.

Short-Term Influence of Recent Trial History on Perceptual Choice Changes with Stimulus Strength

Perceptual decisions, especially for difficult stimuli, can be influenced by choices and outcomes in previous trials. However, it is not well understood how stimulus strength modulates the temporal characteristics as well as the magnitude of trial history influence. We addressed this question using a contrast detection task in freely moving mice. We found that, at lower as compared to higher stimulus contrast, the current choice of the mice was more influenced by choices and outcomes in the past trials and the influence emerged from a longer history. To examine the neural basis of stimulus strength-dependent history influence, we recorded from the secondary motor cortex (M2), a prefrontal region that plays an important role in cue-guided actions and memory-guided behaviors. We found that more M2 neurons conveyed information about choices on the past two trials at lower than at higher contrast. Furthermore, history-trial activity in M2 was important for decoding upcoming choice at low contrast. Thus, trial history influence of perceptual choice is adaptive to the strength of sensory evidence, which may be important for action selection in a dynamic environment.

Dopamine and eye movement control in Parkinson's disease: deficits in corollary discharge signals?

Movement in Parkinson's disease (PD) is fragmented, and the patients depend on visual information in their behavior. This suggests that the patients may have deficits in internally monitoring their own movements. Internal monitoring of movements is assumed to rely on corollary discharge signals that enable the brain to predict the sensory consequences of actions. We studied early-stage PD patients (N = 14), and age-matched healthy control participants (N = 14) to examine whether PD patients reveal deficits in updating their sensory representations after eye movements. The participants performed a double-saccade task where, in order to accurately fixate a second target, the participant must correct for the displacement caused by the first saccade. In line with previous reports, the patients had difficulties in fixating the second target when the eye movement was performed without visual guidance. Furthermore, the patients had difficulties in taking into account the error in the first saccade when making a saccade toward the second target, especially when eye movements were made toward the side with dominant motor symptoms. Across PD patients, the impairments in saccadic eye movements correlated with the integrity of the dopaminergic system as measured with [123I]FP-CIT SPECT: Patients with lower striatal (caudate, anterior putamen, and posterior putamen) dopamine transporter binding made larger errors in saccades. This effect was strongest when patients made memory-guided saccades toward the second target. Our results provide tentative evidence that the motor deficits in PD may be partly due to deficits in internal monitoring of movements.

Remembering rewarding futures: A simulation-selection model of the hippocampus

Despite tremendous progress, the neural circuit dynamics underlying hippocampal mnemonic processing remain poorly understood. We propose a new model for hippocampal function-the simulation-selection model-based on recent experimental findings and neuroecological considerations. Under this model, the mammalian hippocampus evolved to simulate and evaluate arbitrary navigation sequences. Specifically, we suggest that CA3 simulates unexperienced navigation sequences in addition to remembering experienced ones, and CA1 selects from among these CA3-generated sequences, reinforcing those that are likely to maximize reward during offline idling states. High-value sequences reinforced in CA1 may allow flexible navigation toward a potential rewarding location during subsequent navigation. We argue that the simulation-selection functions of the hippocampus have evolved in mammals mostly because of the unique navigational needs of land mammals. Our model may account for why the mammalian hippocampus has evolved not only to remember, but also to imagine episodes, and how this might be implemented in its neural circuits.

Reinforcement learning models of risky choice and the promotion of risk-taking by losses disguised as wins in rats

Risky decisions are inherently characterized by the potential to receive gains or incur losses, and these outcomes have distinct effects on subsequent decision-making. One important factor is that individuals engage in loss-chasing, in which the reception of a loss is followed by relatively increased risk-taking. Unfortunately, the mechanisms of loss-chasing are poorly understood, despite the potential importance for understanding pathological choice behavior. The goal of the present experiment was to illuminate the mechanisms governing individual differences in loss-chasing and risky-choice behaviors. Rats chose between a low-uncertainty outcome that always delivered a variable amount of reward and a high-uncertainty outcome that probabilistically delivered reward. Loss-processing and loss-chasing were assessed in the context of losses disguised as wins (LDWs), which are loss outcomes that are presented along with gain-related stimuli. LDWs have been suggested to interfere with adaptive decision-making in humans and thus potentially increase loss-making. Here, the rats presented with LDWs were riskier, in that they made more choices for the high-uncertainty outcome. A series of nonlinear models were fit to individual rats' data to elucidate the possible psychological mechanisms that best account for individual differences in high-uncertainty choices and loss-chasing behaviors. The models suggested that the rats presented with LDWs were more prone to showing a stay bias following high-uncertainty outcomes compared to rats not presented with LDWs. These results collectively suggest that LDWs acquire conditioned reinforcement properties that encourage continued risk-taking and increase loss-chasing following previous high-risk decisions. (PsycINFO Database Record

Striatal action-value neurons reconsidered

It is generally believed that during economic decisions, striatal neurons represent the values associated with different actions. This hypothesis is based on studies, in which the activity of striatal neurons was measured while the subject was learning to prefer the more rewarding action. Here we show that these publications are subject to at least one of two critical confounds. First, we show that even weak temporal correlations in the neuronal data may result in an erroneous identification of action-value representations. Second, we show that experiments and analyses designed to dissociate action-value representation from the representation of other decision variables cannot do so. We suggest solutions to identifying action-value representation that are not subject to these confounds. Applying one solution to previously identified action-value neurons in the basal ganglia we fail to detect action-value representations. We conclude that the claim that striatal neurons encode action-values must await new experiments and analyses.

Dorsolateral Striatum Engagement Interferes with Early Discrimination Learning

In current models, learning the relationship between environmental stimuli and the outcomes of actions involves both stimulus-driven and goal-directed systems, mediated in part by the DLS and DMS, respectively. However, though these models emphasize the importance of the DLS in governing actions after extensive experience has accumulated, there is growing evidence of DLS engagement from the onset of training. Here, we used in vivo photosilencing to reveal that DLS recruitment interferes with early touchscreen discrimination learning. We also show that the direct output pathway of the DLS is preferentially recruited and causally involved in early learning and find that silencing the normal contribution of the DLS produces plasticity-related alterations in a PL-DMS circuit. These data provide further evidence suggesting that the DLS is recruited in the construction of stimulus-elicited actions that ultimately automate behavior and liberate cognitive resources for other demands, but with a cost to performance at the outset of learning.

Solving the Credit Assignment Problem With the Prefrontal Cortex

In naturalistic multi-cue and multi-step learning tasks, where outcomes of behavior are delayed in time, discovering which choices are responsible for rewards can present a challenge, known as the credit assignment problem. In this review, I summarize recent work that highlighted a critical role for the prefrontal cortex (PFC) in assigning credit where it is due in tasks where only a few of the multitude of cues or choices are relevant to the final outcome of behavior. Collectively, these investigations have provided compelling support for specialized roles of the orbitofrontal (OFC), anterior cingulate (ACC), and dorsolateral prefrontal (dlPFC) cortices in contingent learning. However, recent work has similarly revealed shared contributions and emphasized rich and heterogeneous response properties of neurons in these brain regions. Such functional overlap is not surprising given the complexity of reciprocal projections spanning the PFC. In the concluding section, I overview the evidence suggesting that the OFC, ACC and dlPFC communicate extensively, sharing the information about presented options, executed decisions and received rewards, which enables them to assign credit for outcomes to choices on which they are contingent. This account suggests that lesion or inactivation/inhibition experiments targeting a localized PFC subregion will be insufficient to gain a fine-grained understanding of credit assignment during learning and instead poses refined questions for future research, shifting the focus from focal manipulations to experimental techniques targeting cortico-cortical projections.

Differential coding of reward and movement information in the dorsomedial striatal direct and indirect pathways

The direct and indirect pathways of the basal ganglia have long been thought to mediate behavioral promotion and inhibition, respectively. However, this classic dichotomous model has been recently challenged. To better understand neural processes underlying reward-based learning and movement control, we recorded from direct (dSPNs) and indirect (iSPNs) pathway spiny projection neurons in the dorsomedial striatum of D1-Cre and D2-Cre mice performing a probabilistic Pavlovian conditioning task. dSPNs tend to increase activity while iSPNs decrease activity as a function of reward value, suggesting the striatum represents value in the relative activity levels of dSPNs versus iSPNs. Lick offset-related activity increase is largely dSPN selective, suggesting dSPN involvement in suppressing ongoing licking behavior. Rapid responses to negative outcome and previous reward-related responses are more frequent among iSPNs than dSPNs, suggesting stronger contributions of iSPNs to outcome-dependent behavioral adjustment. These findings provide new insights into striatal neural circuit operations.

Classically the direct and indirect pathways of basal ganglia are understood to have opposing roles in movement and reward learning, but recent work suggests a more complicated view. Here the authors further study indirect and direct pathway neurons, in the context of a probabilistic reward task.

Neural activity in cortico-basal ganglia circuits of juvenile songbirds encodes performance during goal-directed learning

Cortico-basal ganglia circuits are thought to mediate goal-directed learning by a process of outcome evaluation to gradually select appropriate motor actions. We investigated spiking activity in core and shell subregions of the cortical nucleus LMAN during development as juvenile zebra finches are actively engaged in evaluating feedback of self-generated behavior in relation to their memorized tutor song (the goal). Spiking patterns of single neurons in both core and shell subregions during singing correlated with acoustic similarity to tutor syllables, suggesting a process of outcome evaluation. Both core and shell neurons encoded tutor similarity via either increases or decreases in firing rate, although only shell neurons showed a significant association at the population level. Tutor similarity predicted firing rates most strongly during early stages of learning, and shell but not core neurons showed decreases in response variability across development, suggesting that the activity of shell neurons reflects the progression of learning.

Neural Signals Related to Outcome Evaluation Are Stronger in CA1 than CA3

We have shown previously that CA1 conveys significant neural signals necessary to update value of the chosen target, namely chosen value and reward signals. To better understand hippocampal neural processes related to valuation, we compared chosen value- and reward-related neural activity between the CA3 and CA1 regions. Single units were recorded with tetrodes from the dorsal CA3 and CA1 regions of rats performing a dynamic foraging task, and chosen value- and reward-related neural activity was estimated using a reinforcement learning model and multiple regression analyses. Neural signals for chosen value and reward converged in both CA3 and CA1 when a trial outcome was revealed. However, these neural signals were stronger in CA1 than CA3. Consequently, neural signals for reward prediction error and updated chosen value were stronger in CA1 than CA3. Together with our previous finding that CA1 conveys stronger value signals than the subiculum, our results raise the possibility that CA1 might play a particularly important role among hippocampal subregions in evaluating experienced events.

A neuronal activation correlate in striatum and prefrontal cortex of prolonged cocaine intake

Maladaptive changes in the involvement of striatal and frontal cortical regions in drug use are thought to underlie the progression to habitual drug use and loss of cognitive control over drug intake that occur with accumulating drug experience. The present experiments focus on changes in neuronal activity in these regions associated with short-term (10 days) and long-term (60 days) self-administration of cocaine. Quantitative in situ hybridization for the immediate early gene Mkp1 was combined with statistical parametric mapping to assess the distribution of neuronal activity. We hypothesized that neuronal activity in striatum would increase in its dorsal part and that activity in frontal cortex would decrease with prolonged cocaine self-administration experience. Expression of Mkp1 was profoundly increased after cocaine self-administration, and the magnitude of this effect was greater after short-term compared to long-term self-administration. Increased neuronal activity was seen in both dorsal and ventral sectors of the striatum after 10 days exposure to cocaine. However, enhanced activity was restricted to dorsomedial and dorsocentral striatum after 60 days cocaine self-administration. In virtually all medial prefrontal and most orbitofrontal areas, increased expression of Mkp1 was observed after 10 days of cocaine taking, whereas after 60 days, enhanced expression was restricted to caudal parts of medial prefrontal and caudomedial parts of orbitofrontal cortex. Our data reveal functional changes in cellular activity in striatum and frontal cortex with increasing cocaine self-administration experience. These changes might reflect the neural processes that underlie the descent from recreational drug taking to compulsive cocaine use.

Neural substrates underlying effort, time, and risk-based decision making in motivated behavior

All mobile organisms rely on adaptive motivated behavior to overcome the challenges of living in an environment in which essential resources may be limited. A variety of influences ranging from an organism's environment, experiential history, and physiological state all influence a cost-benefit analysis which allows motivation to energize behavior and direct it toward specific goals. Here we review the substantial amount of research aimed at discovering the interconnected neural circuits which allow organisms to carry-out the cost-benefit computations which allow them to behave in adaptive ways. We specifically focus on how the brain deals with different types of costs, including effort requirements, delays to reward and payoff riskiness. An examination of this broad literature highlights the importance of the extended neural circuits which enable organisms to make decisions about these different types of costs. This involves Cortical Structures, including the Anterior Cingulate Cortex (ACC), the Orbital Frontal Cortex (OFC), the Infralimbic Cortex (IL), and prelimbic Cortex (PL), as well as the Baso-Lateral Amygdala (BLA), the Nucleus Accumbens (NAcc), the Ventral Pallidal (VP), the Sub Thalamic Nucleus (STN) among others. Some regions are involved in multiple aspects of cost-benefit computations while the involvement of other regions is restricted to information relating to specific types of costs.

Mesencephalic representations of recent experience influence decision making

Decisions are influenced by recent experience, but the neural basis for this phenomenon is not well understood. Here, we address this question in the context of action selection. We focused on activity in the pedunculopontine tegmental nucleus (PPTg), a mesencephalic region that provides input to several nuclei in the action selection network, in well-trained mice selecting actions based on sensory cues and recent trial history. We found that, at the time of action selection, the activity of many PPTg neurons reflected the action on the previous trial and its outcome, and the strength of this activity predicted the upcoming choice. Further, inactivating the PPTg predictably decreased the influence of recent experience on action selection. These findings suggest that PPTg input to downstream motor regions, where it can be integrated with other relevant information, provides a simple mechanism for incorporating recent experience into the computations underlying action selection.

DOI:http://dx.doi.org/10.7554/eLife.16572.001

The decisions we make are influenced by recent experience, yet it is not known how this experience is represented in the brain. For decisions about when, where and how to move, researchers have hypothesized that recent experience might influence activity in a region of the brainstem – the central trunk of the brain – that is known to be involved in movement.

When deciding when, where and how to move, several areas of the brain are involved in selecting the optimal action. Recent studies suggest that groups of neurons known as locomotor brainstem nuclei may also contribute to making decisions about movements.

Thompson et al. investigated whether a brainstem locomotor area called the pedunculopontine tegmental (PPTg) nucleus in mice might contribute to decision making rather than just conveying the selected response. The mice were trained to recognize particular odors and move to either the left or right to collect a food reward. While the mice were selecting an action, the activity of neurons in the PPTg nucleus reflected the action they had chosen on a previous experience and the outcome of that choice (i.e. whether they received a reward). These representations of past experiences influenced the upcoming decision the mice were about to take.

The findings of Thompson et al. suggest that the PPTg nucleus might play a critical role in the process of selecting the optimal action. Future work will examine what kinds of information about the environment or recent experience have the biggest effect on the activity of this region.

DOI:http://dx.doi.org/10.7554/eLife.16572.002

Neuronal activity in dorsomedial and dorsolateral striatum under the requirement for temporal credit assignment

To investigate neural processes underlying temporal credit assignment in the striatum, we recorded neuronal activity in the dorsomedial and dorsolateral striatum (DMS and DLS, respectively) of rats performing a dynamic foraging task in which a choice has to be remembered until its outcome is revealed for correct credit assignment. Choice signals appeared sequentially, initially in the DMS and then in the DLS, and they were combined with action value and reward signals in the DLS when choice outcome was revealed. Unlike in conventional dynamic foraging tasks, neural signals for chosen value were elevated in neither brain structure. These results suggest that dynamics of striatal neural signals related to evaluating choice outcome might differ drastically depending on the requirement for temporal credit assignment. In a behavioral context requiring temporal credit assignment, the DLS, but not the DMS, might be in charge of updating the value of chosen action by integrating choice, action value, and reward signals together.

A novel strategy for dissecting goal-directed action and arousal components of motivated behavior with a progressive hold-down task

Motivation serves 2 important functions: It guides actions to be goal-directed, and it provides the energy and vigor required to perform the work necessary to meet those goals. Dissociating these 2 processes with existing behavioral assays has been a challenge. In this article, we report a novel experimental strategy to distinguish the 2 processes in mice. First, we characterize a novel motivation assay in which animals must hold down a lever for progressively longer intervals to earn each subsequent reward; we call this the progressive hold-down (PHD) task. We find that performance on the PHD task is sensitive to both food deprivation level and reward value. Next, we use a dose of methamphetamine (METH) 1.0 mg/kg, to evaluate behavior in both the progressive ratio (PR) and PHD tasks. Treatment with METH leads to more persistent lever pressing for food rewards in the PR. In the PHD task, we found that METH increased arousal, which leads to numerous bouts of hyperactive responding but neither increases nor impairs goal-directed action. The results demonstrate that these tools enable a more precise understanding of the underlying processes being altered in manipulations that alter motivated behavior.

Neurophysiology of rule switching in the corticostriatal circuit

The ability to adjust behavioral responses to cues in a changing environment is crucial for survival. Activity in the medial Prefrontal Cortex (mPFC) is thought to both represent rules to guide behavior as well as detect and resolve conflicts between rules in changing contingencies. While lesion and pharmacological studies have supported a crucial role for mPFC in this type of set-shifting, an understanding of how mPFC represents current rules or detects and resolves conflict between different rules is still unclear. Meanwhile, medial dorsal striatum (mDS) receives major projections from mPFC and neural activity of mDS is closely linked to action selection, making the mDS a potential major player for enacting rule-guided action policies. However, exactly what is signaled by mPFC and how this impacts neural signals in mDS is not well known. In this review, we will summarize what is known about neural signals of rules and set shifting in both prefrontal cortex and dorsal striatum, as well as provide questions and directions for future experiments.

Modeling Parkinson's disease falls associated with brainstem cholinergic systems decline

In addition to the primary disease-defining symptoms, approximately half of patients with Parkinson's disease (PD) suffer from postural instability, impairments in gait control and a propensity for falls. Consistent with evidence from patients, we previously demonstrated that combined striatal dopamine (DA) and basal forebrain (BF) cholinergic cell loss causes falls in rats traversing dynamic surfaces. Because evidence suggests that degeneration of brainstem cholinergic neurons arising from the pedunculopontine nucleus (PPN) also contributes to impaired gait and falls, here we assessed the effects of selective cholinergic PPN lesions in combination with striatal DA loss or BF cholinergic cells loss as well as losses in all 3 regions. Results indicate that all combination losses that included the BF cholinergic system slowed traversal and increased slips and falls. However, the performance of rats with losses in all 3 regions (PPN, BF, and DA) was not more severely impaired than following combined BF cholinergic and striatal DA lesions. These results confirm the hypothesis that BF cholinergic-striatal disruption of attentional-motor interactions is a primary source of falls. Additional losses of PPN cholinergic neurons may worsen posture and gait control in situations not captured by the current testing conditions.

Dissociable contribution of nucleus accumbens and dorsolateral striatum to the acquisition of risk choice behavior in the rat

While a growing body of research has suggested that the mesocorticolimbic dopamine systems play a key role in decision making under risk, how the nucleus accumbens (NAC) is involved in the acquisition of risk choice behavior remains unclear. This study used a T-maze task to assess risk-based decision making in which the rat was required to assess the risk by choosing to enter either a small and certain reward arm or a large but uncertain reward arm of the maze. The latter option, when chosen, resulted in provision of 2, 4, or 8 sweeten pellets with a probability (p) of 0.5, 0.25, or 0.125, respectively. Thus the latter arm provided three different conditions of reward ratio, compared to the choice of former arm, which always provided 1 pellet with p=1. This risk choice task was then run with the expected value being equality between the binary choice options. The experimental rats first received an excitoneurotoxic lesion affecting either the NAC or the dorsolateral striatum (DLS) and this was followed by post-lesion behavioral examination. The sham lesion control rats acquired a stable risk choice with regard to each reward ratio over a 10-day test. The pattern of choice behavior appeared in risk-seeking when p=0.5 to obtain 2 pellets, and was risk-averse when larger reward resulted in lower p. The NAC lesion significantly disrupted the acquisition of the aforementioned risk choice behavior and apparently shifted the choice into a risk-averse style for all three reward ratios. No such effect was observed in the rats with DLS lesions. Neither the gross motor action nor the discrimination of different reward magnitudes was impaired by the lesions affecting either the NAC or DLS as assessed by an additional experiment. These findings suggest that firstly there is heterogeneity between NAC and DLS with respect to risk-based decision making, and that secondly the NAC is involved and critical to the acquisition of behavioral choice under risk, specially when the expected value of the reward under the two choice options is equal.

Distinct neural representation in the dorsolateral, dorsomedial, and ventral parts of the striatum during fixed- and free-choice tasks

The striatum is a major input site of the basal ganglia, which play an essential role in decision making. Previous studies have suggested that subareas of the striatum have distinct roles: the dorsolateral striatum (DLS) functions in habitual action, the dorsomedial striatum (DMS) in goal-directed actions, and the ventral striatum (VS) in motivation. To elucidate distinctive functions of subregions of the striatum in decision making, we systematically investigated information represented by phasically active neurons in DLS, DMS, and VS. Rats performed two types of choice tasks: fixed- and free-choice tasks. In both tasks, rats were required to perform nose poking to either the left or right hole after cue-tone presentation. A food pellet was delivered probabilistically depending on the presented cue and the selected action. The reward probability was fixed in fixed-choice task and varied in a block-wise manner in free-choice task. We found the following: (1) when rats began the tasks, a majority of VS neurons increased their firing rates and information regarding task type and state value was most strongly represented in VS; (2) during action selection, information of action and action values was most strongly represented in DMS; (3) action-command information (action representation before action selection) was stronger in the fixed-choice task than in the free-choice task in both DLS and DMS; and (4) action-command information was strongest in DLS, particularly when the same choice was repeated. We propose a hypothesis of hierarchical reinforcement learning in the basal ganglia to coherently explain these results.

Characterizing the associative content of brain structures involved in habitual and goal-directed actions in humans: a multivariate FMRI study

While there is accumulating evidence for the existence of distinct neural systems supporting goal-directed and habitual action selection in the mammalian brain, much less is known about the nature of the information being processed in these different brain regions. Associative learning theory predicts that brain systems involved in habitual control, such as the dorsolateral striatum, should contain stimulus and response information only, but not outcome information, while regions involved in goal-directed action, such as ventromedial and dorsolateral prefrontal cortex and dorsomedial striatum, should be involved in processing information about outcomes as well as stimuli and responses. To test this prediction, human participants underwent fMRI while engaging in a binary choice task designed to enable the separate identification of these different representations with a multivariate classification analysis approach. Consistent with our predictions, the dorsolateral striatum contained information about responses but not outcomes at the time of an initial stimulus, while the regions implicated in goal-directed action selection contained information about both responses and outcomes. These findings suggest that differential contributions of these regions to habitual and goal-directed behavioral control may depend in part on basic differences in the type of information that these regions have access to at the time of decision making.

Modeling falls in Parkinson's disease: Slow gait, freezing episodes and falls in rats with extensive striatal dopamine loss

Falls in patients with Parkinson's disease (PD) are a major and levodopa-unresponsive source of morbidity. We previously described an animal model of falls resulting from impairments in attentional-motor interactions. Reproducing the multisystem dopaminergic-cholinergic cell loss in patients with a history for falls, partial loss of striatal dopamine innervation interacted with loss of forebrain cholinergic neurons to generate falls that was hypothesized to reflect impairments in the attentional control of gait and balance and the sequencing of complex movements [1]. As clinical evidence also indicates that basal ganglia dopamine (DA) loss per se is associated with severe discoordination and thus a greater risk for falls, here we demonstrate that relatively extensive striatal DA loss, in contrast to the lack of effects of smaller, dorsal striatal DA losses and sham lesions, increased falls and slips and caused slowing while traversing dynamic surfaces. Falls in large DA rats were associated specifically with spontaneous or slip-triggered stoppages of forward movement. Collectively, the evidence suggests that low motivation or vigor for movement in general, and for initiating corrective movements in particular, are major sources for falls in rats with large DA losses. Falls are a result of complex cognitive-motor interactions, and rats with large DA losses model the impact of a propensity for freezing of gait when traversing dynamic surfaces.

Nicotinic alteration of decision-making

Addiction to nicotine is characterized by impulses, urges and lack of self-control towards cigarettes. A key element in the process of addiction is the development of habits oriented towards nicotine consumption that surpass flexible systems as a consequence of a gradual adaptation to chronic drug exposure. However, the long-term effects of nicotine on brain circuits also induce wide changes in decision-making processes, affecting behaviors unrelated to cigarettes. This review aims at providing an update on the implications of nicotine on general decision-making processes, with an emphasis on impulsivity and risk-taking. As impulsivity is a rather ambiguous behavioral trait, we build on economic and normative theories to better characterize these nicotine-induced alterations in decision-making. Nonetheless, experimental data are sparse and often contradictory. We will discuss how the latest findings on the neurobiological basis of choice behavior may help disentangling these issues. We focus on the role of nicotine acetylcholine receptors and their different subunits, and on the spatio-temporal dynamics (i.e. diversity of the neural circuits, short- and long-term effects) of both endogenous acetylcholine and nicotine action. Finally, we try to link these neurobiological results with neuro-computational models of attention, valuation and action, and of the role of acetylcholine in these decision processes. This article is part of the Special Issue entitled 'The Nicotinic Acetylcholine Receptor: From Molecular Biology to Cognition'.

Interpretive monitoring in the caudate nucleus

In a dynamic environment an organism has to constantly adjust ongoing behavior to adapt to a given context. This process requires continuous monitoring of ongoing behavior to provide its meaningful interpretation. The caudate nucleus is known to have a role in behavioral monitoring, but the nature of these signals during dynamic behavior is still unclear. We recorded neuronal activity in the caudate nucleus in monkeys during categorization behavior that changed rapidly across contexts. We found that neuronal activity maintained representation of the identity and context of a recently categorized stimulus, as well as interpreted the behavioral meaningfulness of the maintained trace. The accuracy of this cognitive monitoring signal was highest for behavior for which subjects were prone to make errors. Thus, the caudate nucleus provides interpretive monitoring of ongoing behavior, which is necessary for contextually specific decisions to adapt to rapidly changing conditions.

DOI:http://dx.doi.org/10.7554/eLife.03727.001

The ability to adapt behavior in a changing environment is a hallmark of intelligent systems. From adjusting our driving speed to match road conditions to responding to a last-minute change of plans, mental flexibility underpins much of our day-to-day functioning.

To perform optimally, an animal must continuously monitor its own behavior and adjust it according to circumstances. A region of the brain called the caudate nucleus is thought to contribute to this process by keeping track of the relation between an action and its outcomes, but it is not clear how it monitors cognitive aspects of ongoing behavior.

Yanike and Ferrera have clarified this process by recording electrical activity from the caudate nucleus in two monkeys as they categorized visual stimuli. The monkeys viewed a moving stimulus and classified it as ‘fast’ or ‘slow’ relative to a reference speed that varied from trial to trial. The monkeys were trained to use two different references speeds and were told which reference speed to use at the start of each trial. They used an eye movement to indicate their decision.

Most neurons within the caudate nucleus responded after the monkey had made a decision, suggesting that these neurons might be involved in evaluating the decision that had just been made. The response of the neurons depended on the stimulus speed, and also on the category (fast or slow) in which the stimulus belonged. This observation indicates that the caudate nucleus tracked the context (reference speed) as well as the stimulus speed.

Yanike and Ferrera also showed that the response of the entire population of caudate neurons could be decoded to reveal both the speed of the stimulus and whether the monkey had categorized it as fast or slow. This shows that after a decision has been made, neurons continue to signal both the stimulus and the context in which that stimulus was presented. Such ‘post-decision’ monitoring is important for anticipating the outcome of the decision. Overall the results suggest that the caudate nucleus helps animals to adapt their behavior to rapidly changing circumstances by supporting decision-making that takes context into account.

DOI:http://dx.doi.org/10.7554/eLife.03727.002

Dopamine depletion in either the dorsomedial or dorsolateral striatum impairs egocentric Cincinnati water maze performance while sparing allocentric Morris water maze learning

Both egocentric route-based learning and spatial learning, as assessed by the Cincinnati water maze (CWM) and Morris water maze (MWM), respectively, are impaired following an 80% dopamine (DA) loss in the neostriatum after 6-hydroxydopamine (6-OHDA) administration in rats. The dorsolateral striatum (DLS) and the dorsomedial striatum (DMS) are implicated in different navigational learning types, namely the DLS is implicated in egocentric learning while the DMS is implicated in spatial learning. This experiment tested whether selective DA loss through 6-OHDA lesions in the DMS or DLS would impair one or both types of navigation. Both DLS and DMS DA loss significantly impaired route-based CWM learning, without affecting spatial or cued MWM performance. DLS 6-OHDA lesions produced a 75% DA loss in this region, with no changes in other monoamine levels in the DLS or DMS. DMS 6-OHDA lesions produced a 62% DA loss in this region, without affecting other monoamine levels in the DMS or DLS. The results indicate a role for DA in DLS and DMS regions in route-based egocentric but not spatial learning and memory. Spatial learning deficits may require more pervasive monoamine reductions within each region before deficits are exhibited. This is the first study to implicate DLS and DMS DA in route-based egocentric navigation.

Oscillatory activity in the medial prefrontal cortex and nucleus accumbens correlates with impulsivity and reward outcome

Actions expressed prematurely without regard for their consequences are considered impulsive. Such behaviour is governed by a network of brain regions including the prefrontal cortex (PFC) and nucleus accumbens (NAcb) and is prevalent in disorders including attention deficit hyperactivity disorder (ADHD) and drug addiction. However, little is known of the relationship between neural activity in these regions and specific forms of impulsive behaviour. In the present study we investigated local field potential (LFP) oscillations in distinct sub-regions of the PFC and NAcb on a 5-choice serial reaction time task (5-CSRTT), which measures sustained, spatially-divided visual attention and action restraint. The main findings show that power in gamma frequency (50-60 Hz) LFP oscillations transiently increases in the PFC and NAcb during both the anticipation of a cue signalling the spatial location of a nose-poke response and again following correct responses. Gamma oscillations were coupled to low-frequency delta oscillations in both regions; this coupling strengthened specifically when an error response was made. Theta (7-9 Hz) LFP power in the PFC and NAcb increased during the waiting period and was also related to response outcome. Additionally, both gamma and theta power were significantly affected by upcoming premature responses as rats waited for the visual cue to respond. In a subgroup of rats showing persistently high levels of impulsivity we found that impulsivity was associated with increased error signals following a nose-poke response, as well as reduced signals of previous trial outcome during the waiting period. Collectively, these in-vivo neurophysiological findings further implicate the PFC and NAcb in anticipatory impulsive responses and provide evidence that abnormalities in the encoding of rewarding outcomes may underlie trait-like impulsive behaviour.

Neural correlates of strategic reasoning during competitive games

Although human and animal behaviors are largely shaped by reinforcement and punishment, choices in social settings are also influenced by information about the knowledge and experience of other decision-makers. During competitive games, monkeys increased their payoffs by systematically deviating from a simple heuristic learning algorithm and thereby countering the predictable exploitation by their computer opponent. Neurons in the dorsomedial prefrontal cortex (dmPFC) signaled the animal's recent choice and reward history that reflected the computer's exploitative strategy. The strength of switching signals in the dmPFC also correlated with the animal's tendency to deviate from the heuristic learning algorithm. Therefore, the dmPFC might provide control signals for overriding simple heuristic learning algorithms based on the inferred strategies of the opponent.

Where attention falls: Increased risk of falls from the converging impact of cortical cholinergic and midbrain dopamine loss on striatal function

Falls are a major source of hospitalization, long-term institutionalization, and death in older adults and patients with Parkinson's disease (PD). Limited attentional resources are a major risk factor for falls. In this review, we specify cognitive-behavioral mechanisms that produce falls and map these mechanisms onto a model of multi-system degeneration. Results from PET studies in PD fallers and findings from a recently developed animal model support the hypothesis that falls result from interactions between loss of basal forebrain cholinergic projections to the cortex and striatal dopamine loss. Striatal dopamine loss produces inefficient, low-vigor gait, posture control, and movement. Cortical cholinergic deafferentation impairs a wide range of attentional processes, including monitoring of gait, posture and complex movements. Cholinergic cell loss reveals the full impact of striatal dopamine loss on motor performance, reflecting loss of compensatory attentional supervision of movement. Dysregulation of dorsomedial striatal circuitry is an essential, albeit not exclusive, mediator of falls in this dual-system model. Because cholinergic neuromodulatory activity influences cortical circuitry primarily via stimulation of α4β2* nicotinic acetylcholine receptors, and because agonists at these receptors are known to benefit attentional processes in animals and humans, treating PD fallers with such agonists, as an adjunct to dopaminergic treatment, is predicted to reduce falls. Falls are an informative behavioral endpoint to study attentional-motor integration by striatal circuitry.

Representation of outcome risk and action in the anterior caudate nucleus

The anterior caudate nucleus is essential for goal-directed behavior because it links outcome information to actions. It is well known that caudate neurons provide a variety of reward-related and action signals. However, it is still unclear how the two signals are integrated. We investigated whether and how outcome risk modulates spatial representation. We recorded neural activity in the anterior caudate nucleus while monkeys made saccades to multiple spatial targets, each associated with either fixed (safe) or variable (risky) amount of reward. We report that individual neurons combined the outcome reward signal with spatial information about the direction of saccades. These signals could be reliably read out from the populations of neurons. Moreover, the prospect of a risky outcome improved the quality of spatial information. These results provide direct evidence that global spatial representation in the caudate is modulated by outcome, which can be important for flexible control of behavior, particularly during learning and habit formation, when outcomes vary.

Neural correlates of temporal credit assignment in the parietal lobe

Empirical studies of decision making have typically assumed that value learning is governed by time, such that a reward prediction error arising at a specific time triggers temporally-discounted learning for all preceding actions. However, in natural behavior, goals must be acquired through multiple actions, and each action can have different significance for the final outcome. As is recognized in computational research, carrying out multi-step actions requires the use of credit assignment mechanisms that focus learning on specific steps, but little is known about the neural correlates of these mechanisms. To investigate this question we recorded neurons in the monkey lateral intraparietal area (LIP) during a serial decision task where two consecutive eye movement decisions led to a final reward. The underlying decision trees were structured such that the two decisions had different relationships with the final reward, and the optimal strategy was to learn based on the final reward at one of the steps (the “F” step) but ignore changes in this reward at the remaining step (the “I” step). In two distinct contexts, the F step was either the first or the second in the sequence, controlling for effects of temporal discounting. We show that LIP neurons had the strongest value learning and strongest post-decision responses during the transition after the F step regardless of the serial position of this step. Thus, the neurons encode correlates of temporal credit assignment mechanisms that allocate learning to specific steps independently of temporal discounting.

Modeling fall propensity in Parkinson's disease: deficits in the attentional control of complex movements in rats with cortical-cholinergic and striatal-dopaminergic deafferentation

Cognitive symptoms, complex movement deficits, and increased propensity for falls are interrelated and levodopa-unresponsive symptoms in patients with Parkinson's disease (PD). We developed a test system for the assessment of fall propensity in rats and tested the hypothesis that interactions between loss of cortical cholinergic and striatal dopaminergic afferents increase fall propensity. Rats were trained to traverse stationary and rotating rods, placed horizontally or at inclines, and while exposed to distractors. Rats also performed an operant Sustained Attention Task (SAT). Partial cortical cholinergic and/or caudate dopaminergic deafferentation were produced by bilateral infusions of 192 IgG-saporin (SAP) into the basal forebrain and/or 6-hydroxydopamine (6-OHDA) into the caudate nucleus, respectively, modeling the lesions seen in early PD. Rats with dual cholinergic-dopaminergic lesions (DL) fell more frequently than SAP or 6-OHDA rats. Falls in DL rats were associated with incomplete rebalancing after slips and low traversal speed. Ladder rung walking and pasta handling performance did not indicate sensorimotor deficits. SAT performance was impaired in DL and SAP rats; however, SAT performance and falls were correlated only in DL rats. Furthermore, in DL rats, but not in rats with only dopaminergic lesions, the placement and size of dopaminergic lesion correlated significantly with fall rates. The results support the hypothesis that after dual cholinergic-dopaminergic lesions, attentional resources can no longer be recruited to compensate for diminished striatal control of complex movement, thereby "unmasking" impaired striatal control of complex movements and yielding falls.

Cortical signals for rewarded actions and strategic exploration

In stable environments, decision makers can exploit their previously learned strategies for optimal outcomes, while exploration might lead to better options in unstable environments. Here, to investigate the cortical contributions to exploratory behavior, we analyzed single-neuron activity recorded from four different cortical areas of monkeys performing a matching-pennies task and a visual search task, which encouraged and discouraged exploration, respectively. We found that neurons in multiple regions in the frontal and parietal cortex tended to encode signals related to previously rewarded actions more reliably than unrewarded actions. In addition, signals for rewarded choices in the supplementary eye field were attenuated during the visual search task and were correlated with the tendency to switch choices during the matching-pennies task. These results suggest that the supplementary eye field might play a unique role in encouraging animals to explore alternative decision-making strategies.