Functional properties of the basal ganglia's re-entrant loop architecture: selection and reinforcement

Dendritic Release of Neurotransmitters

Release of neuroactive substances by exocytosis from dendrites is surprisingly widespread and is not confined to a particular class of transmitters: it occurs in multiple brain regions, and includes a range of neuropeptides, classical neurotransmitters, and signaling molecules, such as nitric oxide, carbon monoxide, ATP, and arachidonic acid. This review is focused on hypothalamic neuroendocrine cells that release vasopressin and oxytocin and midbrain neurons that release dopamine. For these two model systems, the stimuli, mechanisms, and physiological functions of dendritic release have been explored in greater detail than is yet available for other neurons and neuroactive substances. © 2017 American Physiological Society. Compr Physiol 7:235-252, 2017.

Competitive Disinhibition Mediates Behavioral Choice and Sequences in Drosophila

Even a simple sensory stimulus can elicit distinct innate behaviors and sequences. During sensorimotor decisions, competitive interactions among neurons that promote distinct behaviors must ensure the selection and maintenance of one behavior, while suppressing others. The circuit implementation of these competitive interactions is still an open question. By combining comprehensive electron microscopy reconstruction of inhibitory interneuron networks, modeling, electrophysiology, and behavioral studies, we determined the circuit mechanisms that contribute to the Drosophila larval sensorimotor decision to startle, explore, or perform a sequence of the two in response to a mechanosensory stimulus. Together, these studies reveal that, early in sensory processing, (1) reciprocally connected feedforward inhibitory interneurons implement behavioral choice, (2) local feedback disinhibition provides positive feedback that consolidates and maintains the chosen behavior, and (3) lateral disinhibition promotes sequence transitions. The combination of these interconnected circuit motifs can implement both behavior selection and the serial organization of behaviors into a sequence.

Principles of Synaptic Organization of GABAergic Interneurons in the Striatum

The striatum, the entry nucleus of the basal ganglia, lacks laminar or columnar organization of its principal cells; nevertheless, functional data suggest that it is spatially organized. Here we examine whether the connectivity and synaptic organization of striatal GABAergic interneurons contributes to such spatial organization. Focusing on the two main classes of striatal GABAergic interneurons (fast-spiking interneurons [FSIs] and low-threshold-spiking interneurons [LTSIs]), we apply a combination of optogenetics and viral tracing approaches to dissect striatal microcircuits in mice. Our results reveal fundamental differences between the synaptic organizations of both interneuron types. FSIs target exclusively striatal projection neurons (SPNs) within close proximity and form strong synapses on the proximal somatodendritic region. In contrast, LTSIs target both SPNs and cholinergic interneurons, and synaptic connections onto SPNs are made exclusively over long distances and onto distal dendrites. These results suggest fundamentally different functions of FSIs and LTSIs in shaping striatal output.

Action Experience and Action Discovery in Medicated Individuals with Parkinson's Disease

Parkinson’s disease (PD) is a neurodegenerative disorder that markedly affects voluntary action. While regular dopamine treatment can help restore motor function, dopamine also influences cognitive portions of the action system. Previous studies have demonstrated that dopamine medication boosts action-effect associations, which are crucial for the discovery of new voluntary actions. In the present study, we investigated whether neural processes involved in the discovery of new actions are altered in PD participants on regular dopamine treatment, compared to healthy age-matched controls. We recorded brain electroencephalography (EEG) activity while PD patients and age-matched controls performed action discovery (AD) and action control tasks. We found that the novelty P3, a component normally present when there is uncertainty about the occurrence of the sensory effect, was enhanced in PD patients. However, AD was maintained in PD patients, and the novelty P3 demonstrated normal learning-related reductions. Crucially, we found that in PD patients the causal association between an action and its resulting sensory outcome did not modulate the amplitude of the feedback correct-related positivity (fCRP), an EEG component sensitive to the association between an action and its resulting effect. Collectively, these preliminary results suggest that the formation of long-term action-outcome representations may be maintained in PD patients on regular dopamine treatment, but the initial experience of action-effect association may be affected.

The Dopamine Receptor D5 May Influence Age of Onset: An Exploratory Study on Indo-Caucasoid ADHD Subjects

The objective was to investigate contribution of the dopamine receptor 5 (DRD5) gene variants in the symptoms of attention-deficit/hyperactivity disorder (ADHD) probands since brain regions identified to be affected in these group of patients have higher expression of the DRD5 receptor. Out of 22 exonic variants, 19 were monomorphic in the Indo-Caucasoid individuals. rs6283 "C" and rs113828117 "A" exhibited significant higher occurrence in families with ADHD probands. Several haplotypes showed biased occurrence in the probands. Early and late onset groups exhibited significantly different genotypic frequencies. A new G>A substitution was observed in the control samples only. The late onset group exhibited higher scores for hyperactivity as compared to the early onset group. The authors infer that the age of onset of ADHD may at least partially be affected by DRD5 variants warranting further investigation on the role of DRD5 in the disease etiology.

Resting-state connectivity predicts levodopa-induced dyskinesias in Parkinson's disease

Background

Levodopa‐induced dyskinesias are a common side effect of dopaminergic therapy in PD, but their neural correlates remain poorly understood.

Objectives

This study examines whether dyskinesias are associated with abnormal dopaminergic modulation of resting‐state cortico‐striatal connectivity.

Methods

Twelve PD patients with peak‐of‐dose dyskinesias and 12 patients without dyskinesias were withdrawn from dopaminergic medication. All patients received a single dose of fast‐acting soluble levodopa and then underwent resting‐state functional magnetic resonance imaging before any dyskinesias emerged. Levodopa‐induced modulation of cortico‐striatal resting‐state connectivity was assessed between the putamen and the following 3 cortical regions of interest: supplementary motor area, primary sensorimotor cortex, and right inferior frontal gyrus. These functional connectivity measures were entered into a linear support vector classifier to predict whether an individual patient would develop dyskinesias after levodopa intake. Linear regression analysis was applied to test which connectivity measures would predict dyskinesia severity.

Results

Dopaminergic modulation of resting‐state connectivity between the putamen and primary sensorimotor cortex in the most affected hemisphere predicted whether patients would develop dyskinesias with a specificity of 100% and a sensitivity of 91% (P < .0001). Modulation of resting‐state connectivity between the supplementary motor area and putamen predicted interindividual differences in dyskinesia severity (R² = 0.627, P = .004). Resting‐state connectivity between the right inferior frontal gyrus and putamen neither predicted dyskinesia status nor dyskinesia severity.

Conclusions

The results corroborate the notion that altered dopaminergic modulation of cortico‐striatal connectivity plays a key role in the pathophysiology of dyskinesias in PD. © 2016 International Parkinson and Movement Disorder Society

Somatodendritic dopamine release: recent mechanistic insights

Dopamine (DA) is a key transmitter in motor, reward and cogitative pathways, with DA dysfunction implicated in disorders including Parkinson's disease and addiction. Located in midbrain, DA neurons of the substantia nigra pars compacta project via the medial forebrain bundle to the dorsal striatum (caudate putamen), and DA neurons in the adjacent ventral tegmental area project to the ventral striatum (nucleus accumbens) and prefrontal cortex. In addition to classical vesicular release from axons, midbrain DA neurons exhibit DA release from their cell bodies and dendrites. Somatodendritic DA release leads to activation of D2 DA autoreceptors on DA neurons that inhibit their firing via G-protein-coupled inwardly rectifying K(+) channels. This helps determine patterns of DA signalling at distant axonal release sites. Somatodendritically released DA also acts via volume transmission to extrasynaptic receptors that modulate local transmitter release and neuronal activity in the midbrain. Thus, somatodendritic release is a pivotal intrinsic feature of DA neurons that must be well defined in order to fully understand the physiology and pathophysiology of DA pathways. Here, we review recent mechanistic aspects of somatodendritic DA release, with particular emphasis on the Ca(2+) dependence of release and the potential role of exocytotic proteins.

Task complexity and location specific changes of cortical thickness in executive and salience networks after working memory training

Novel activities and experiences shape the brain's structure and organisation and, hence, our behaviour. However, evidence from structural plasticity studies remains mixed and the neural correlates of learning and practice are still poorly understood. We conducted a robustly designed study into grey matter plasticity following 2 months of working memory training. We generated a priori hypotheses regarding the location of plastic effects across three cognitive control networks (executive, anterior salience and basal ganglia networks), and compared the effects of adaptive training (n=20) with a well-matched active control group (n=20) which differed in training complexity and included extensive cognitive assessment before and after the training. Adaptive training relative to control activities resulted in a complex pattern of subtle and localised structural changes: Training was associated with increases in cortical thickness in right-lateralised executive regions, notably the right caudal middle frontal cortex, as well as increases in the volume of the left pallidum. In addition the training group showed reductions of thickness in the right insula, which were correlated with training-induced improvements in backward digit span performance. Unexpectedly, control activities were associated with reductions in thickness in the right pars triangularis. These results suggest that the direction of activity-induced plastic changes depend on the level of training complexity as well as brain location. These observations are consistent with the view that the brain responds dynamically to environmental demands by focusing resources on task relevant networks and eliminating irrelevant processing for the purpose of energy reduction.

Inability to acquire spatial information and deploy spatial search strategies in mice with lesions in dorsomedial striatum

Dorsal striatum has been shown to contribute to spatial learning and memory, but the role of striatal subregions in this important aspect of cognitive functioning remains unclear. Moreover, the spatial-cognitive mechanisms that underlie the involvement of these regions in spatial navigation have scarcely been studied. We therefore compared spatial learning and memory performance in mice with lesions in dorsomedial (DMS) and dorsolateral striatum (DLS) using the hidden-platform version of the Morris water maze (MWM) task. Compared to sham-operated controls, animals with DMS damage were impaired during MWM acquisition training. These mice displayed delayed spatial learning, increased thigmotaxis, and increased search distance to the platform, in the absence of major motor dysfunction, working memory defects or changes in anxiety or exploration. They failed to show a preference for the target quadrant during probe trials, which further indicates that spatial reference memory was impaired in these animals. Search strategy analysis moreover demonstrated that DMS-lesioned mice were unable to deploy cognitively advanced spatial search strategies. Conversely, MWM performance was barely affected in animals with lesions in DLS. In conclusion, our results indicate that DMS and DLS display differential functional involvement in spatial learning and memory. Our results show that DMS, but not DLS, is crucial for the ability of mice to acquire spatial information and their subsequent deployment of spatial search strategies. These data clearly identify DMS as a crucial brain structure for spatial learning and memory, which could explain the occurrence of neurocognitive impairments in brain disorders that affect the dorsal striatum.

Complex sensorimotor transformation processes required for response selection are facilitated by the striatum

Both fronto-parietal networks and the basal ganglia play an important role in action cascading. It is well-known that cortical structures mediate sensorimotor transformation for this purpose. The striatum receives extensive input from those cortical structures and has been shown to be modulated by the predictability of cortical input. Until today, it has however remained unclear whether the processing of spatial codes or even sensorimotor transformation processes for the purpose of action cascading involve the striatum. We therefore examined this question by means of fMRI using a stop-change task that varied the predictability as well as the complexity of sensorimotor transformations required for correct responding in the context of action cascading. On the behavioral level, we found that the complexity of sensorimotor transformation processes only prolonged reaction times when the requirement for this transformation was predictable. fMRI results matched this effect showing enhanced activity of the caudate in case a complex sensorimotor transformation could be anticipated. Irrespective of the complexity of the required transformations, the putamen was furthermore involved in the prediction of imminent action cascading demands. Taken together, our findings give rise to a conceptual advance regarding basal ganglia function by showing that the anticipation and, more importantly, processing of complex sensorimotor transformation processes involves the striatum.

Early use of oleanolic acid provides protection against 6-hydroxydopamine induced dopamine neurodegeneration

Oleanolic acid is a triterpenoid that has been shown to possess antioxidant properties. In this study we investigated the effects of oleanolic acid in a parkinsonian rat model. Unilateral 6-hydroxydopamine (6-OHDA) lesions were carried out on postnatal day (PND) 60 in 4 groups viz. (1) Rats that started oleanolic acid treatment 7 days prior to lesion. (2) Rats not treated with oleanolic acid. (3) Rats that started oleanolic acid treatment 1 day post-lesion. (4) Rats treated with oleanolic acid 7 days post-lesion. The degree of forelimb impairment was assessed using limb use asymmetry and forelimb akinesia tests. Neurochemical changes were assessed using a Dopamine ELISA kit and mitochondrial apoptosis was measured using a mitochondrial apoptosis detection kit. In this study, animals injected with 6-OHDA displayed forelimb use asymmetry that was ameliorated by treatment with oleanolic acid 7 days pre- and 1 day post-lesion. In the cylinder test, rats injected with 6-OHDA favored using the forelimb ipsilateral (unimpaired) to the lesioned hemisphere while rats treated with oleanolic acid used the forelimb contralateral (impaired) to the lesioned hemisphere significantly more. Rats treated with oleanolic acid 7 days pre- and 1 day post-lesion had more dopamine in the striatum than the non-treated or the 7 days after lesion rats. Similarly, 6-OHDA-induced membrane depolarization was decreased in rats that received oleanolic acid treatment pre- or immediately post-lesion. This suggests that early treatment with oleanolic acid protects dopamine neurons from the toxic effects of 6-OHDA.

A cortical-subcortical syntax pathway linking Broca's area and the striatum

Combinatorial syntax has been shown to be underpinned by cortical key regions such as Broca's area and temporal cortices, and by subcortical structures such as the striatum. The cortical regions are connected via several cortico-to-cortical tracts impacting syntactic processing (e.g., the arcuate) but it remains unclear whether and how the striatum can be integrated into this cortex-centered syntax network. Here, we used a systematic stepwise approach to investigate the existence and syntactic function of an additional deep Broca-striatum pathway. We first asked 15 healthy controls and 12 patients with frontal/striatal lesions to perform three syntax tests. The results obtained were subjected to voxel-based lesion-symptom mapping (VLSM) to provide an anatomo-functional approximation of the pathway. The significant VLSM clusters were then overlapped with the probability maps of four cortico-cortical language tracts generated for 12 healthy participants (arcuate, extreme capsule fiber system, uncinate, aslant), including a probabilistic Broca-striatum tract. Finally, we carried out quantitative analyses of the relationship between the lesion load along the tracts and syntactic processing, by calculating tract-lesion overlap for each patient and analyzing the correlation with syntactic data. Our findings revealed a Broca-striatum tract linking BA45 with the left caudate head and overlapping with VLSM voxel clusters relating to complex syntax. The lesion load values for this tract were correlated with complex syntax scores, whereas no such correlation was observed for the other tracts. These results extend current syntax-network models, by adding a deep "Broca-caudate pathway," and are consistent with functional accounts of frontostriatal circuits.

Abnormal dopaminergic modulation of striato-cortical networks underlies levodopa-induced dyskinesias in humans

Dopaminergic signalling in the striatum contributes to reinforcement of actions and motivational enhancement of motor vigour. Parkinson's disease leads to progressive dopaminergic denervation of the striatum, impairing the function of cortico-basal ganglia networks. While levodopa therapy alleviates basal ganglia dysfunction in Parkinson's disease, it often elicits involuntary movements, referred to as levodopa-induced peak-of-dose dyskinesias. Here, we used a novel pharmacodynamic neuroimaging approach to identify the changes in cortico-basal ganglia connectivity that herald the emergence of levodopa-induced dyskinesias. Twenty-six patients with Parkinson's disease (age range: 51-84 years; 11 females) received a single dose of levodopa and then performed a task in which they had to produce or suppress a movement in response to visual cues. Task-related activity was continuously mapped with functional magnetic resonance imaging. Dynamic causal modelling was applied to assess levodopa-induced modulation of effective connectivity between the pre-supplementary motor area, primary motor cortex and putamen when patients suppressed a motor response. Bayesian model selection revealed that patients who later developed levodopa-induced dyskinesias, but not patients without dyskinesias, showed a linear increase in connectivity between the putamen and primary motor cortex after levodopa intake during movement suppression. Individual dyskinesia severity was predicted by levodopa-induced modulation of striato-cortical feedback connections from putamen to the pre-supplementary motor area (Pcorrected = 0.020) and primary motor cortex (Pcorrected = 0.044), but not feed-forward connections from the cortex to the putamen. Our results identify for the first time, aberrant dopaminergic modulation of striatal-cortical connectivity as a neural signature of levodopa-induced dyskinesias in humans. We argue that excessive striato-cortical connectivity in response to levodopa produces an aberrant reinforcement signal producing an abnormal motor drive that ultimately triggers involuntary movements.

Multisensory training reverses midbrain lesion-induced changes and ameliorates haemianopia

Failure to attend to visual cues is a common consequence of visual cortex injury. Here, we report on a behavioural strategy whereby cross-modal (auditory-visual) training reinstates visuomotor competencies in animals rendered haemianopic by complete unilateral visual cortex ablation. The re-emergence of visual behaviours is correlated with the reinstatement of visual responsiveness in deep layer neurons of the ipsilesional superior colliculus (SC). This functional recovery is produced by training-induced alterations in descending influences from association cortex that allowed these midbrain neurons to once again transform visual cues into appropriate orientation behaviours. The findings underscore the inherent plasticity and functional breadth of phylogenetically older visuomotor circuits that can express visual capabilities thought to have been subsumed by more recently evolved brain regions. These observations suggest the need for reevaluating current concepts of functional segregation in the visual system and have important implications for strategies aimed at ameliorating trauma-induced visual deficits in humans.

Basal ganglia circuit loops, dopamine and motivation: A review and enquiry

Dopamine neurons located in the midbrain play a role in motivation that regulates approach behavior (approach motivation). In addition, activation and inactivation of dopamine neurons regulate mood and induce reward and aversion, respectively. Accumulating evidence suggests that such motivational role of dopamine neurons is not limited to those located in the ventral tegmental area, but also in the substantia nigra. The present paper reviews previous rodent work concerning dopamine's role in approach motivation and the connectivity of dopamine neurons, and proposes two working models: One concerns the relationship between extracellular dopamine concentration and approach motivation. High, moderate and low concentrations of extracellular dopamine induce euphoric, seeking and aversive states, respectively. The other concerns circuit loops involving the cerebral cortex, basal ganglia, thalamus, epithalamus, and midbrain through which dopaminergic activity alters approach motivation. These models should help to generate hypothesis-driven research and provide insights for understanding altered states associated with drugs of abuse and affective disorders.

Improving response inhibition in Parkinson's disease with atomoxetine

BACKGROUND

Dopaminergic drugs remain the mainstay of Parkinson's disease therapy but often fail to improve cognitive problems such as impulsivity. This may be due to the loss of other neurotransmitters, including noradrenaline, which is linked to impulsivity and response inhibition. We therefore examined the effect of the selective noradrenaline reuptake inhibitor atomoxetine on response inhibition in a stop-signal paradigm.

METHODS

This pharmacological functional magnetic resonance imaging study used a double-blinded randomized crossover design with low-frequency inhibition trials distributed among frequent Go trials. Twenty-one patients received 40 mg atomoxetine or placebo. Control subjects were tested on no-drug. The effects of disease and drug on behavioral performance, regional brain activity, and functional connectivity were analyzed using general linear models. Anatomical connectivity was examined using diffusion-weighted imaging.

RESULTS

Patients with Parkinson's disease had longer stop-signal reaction times, less stop-related activation in the right inferior frontal gyrus (RIFG), and weaker functional connectivity between the RIFG and striatum compared with control subjects. Atomoxetine enhanced stop-related RIFG activation in proportion to disease severity. Although there was no overall behavioral benefit from atomoxetine, analyses of individual differences revealed that enhanced response inhibition by atomoxetine was associated with increased RIFG activation and functional frontostriatal connectivity. Improved performance was more likely in patients with higher structural frontostriatal connectivity.

CONCLUSIONS

This study suggests that enhanced prefrontal cortical activation and frontostriatal connectivity by atomoxetine may improve response inhibition in Parkinson's disease. These results point the way to new stratified clinical trials of atomoxetine to treat impulsivity in selected patients with Parkinson's disease.

Intrinsic motivations drive learning of eye movements: an experiment with human adults

Intrinsic motivations drive the acquisition of knowledge and skills on the basis of novel or surprising stimuli or the pleasure to learn new skills. In so doing, they are different from extrinsic motivations that are mainly linked to drives that promote survival and reproduction. Intrinsic motivations have been implicitly exploited in several psychological experiments but, due to the lack of proper paradigms, they are rarely a direct subject of investigation. This article investigates how different intrinsic motivation mechanisms can support the learning of visual skills, such as "foveate a particular object in space", using a gaze contingency paradigm. In the experiment participants could freely foveate objects shown in a computer screen. Foveating each of two "button" pictures caused different effects: one caused the appearance of a simple image (blue rectangle) in unexpected positions, while the other evoked the appearance of an always-novel picture (objects or animals). The experiment studied how two possible intrinsic motivation mechanisms might guide learning to foveate one or the other button picture. One mechanism is based on the sudden, surprising appearance of a familiar image at unpredicted locations, and a second one is based on the content novelty of the images. The results show the comparative effectiveness of the mechanism based on image novelty, whereas they do not support the operation of the mechanism based on the surprising location of the image appearance. Interestingly, these results were also obtained with participants that, according to a post experiment questionnaire, had not understood the functions of the different buttons suggesting that novelty-based intrinsic motivation mechanisms might operate even at an unconscious level.

A perspective on neural and cognitive mechanisms of error commission

Behavioral adaptation and cognitive control are crucial for goal-reaching behaviors. Every creature is ubiquitously faced with choices between behavioral alternatives. Common sense suggests that errors are an important source of information in the regulation of such processes. Several theories exist regarding cognitive control and the processing of undesired outcomes. However, most of these models focus on the consequences of an error, and less attention has been paid to the mechanisms that underlie the commissioning of an error. In this article, we present an integrative review of neuro-cognitive models that detail the determinants of the occurrence of response errors. The factors that may determine the likelihood of committing errors are likely related to the stability of task-representations in prefrontal networks, attentional selection mechanisms and mechanisms of action selection in basal ganglia circuits. An important conclusion is that the likelihood of committing an error is not stable over time but rather changes depending on the interplay of different functional neuro-anatomical and neuro-biological systems. We describe factors that might determine the time-course of cognitive control and the need to adapt behavior following response errors. Finally, we outline the mechanisms that may proof useful for predicting the outcomes of cognitive control and the emergence of response errors in future research.

Inhibitory and excitatory neuromodulation by hydrogen peroxide: translating energetics to information

Historically, brain neurochemicals have been broadly classified as energetic or informational. However, increasing evidence implicates metabolic substrates and byproducts as signalling agents, which blurs the boundary between energy and information, and suggests the introduction of a new category for 'translational' substances that convey changes in energy state to information. One intriguing example is hydrogen peroxide (H2 O2 ), which is a small, readily diffusible molecule. Produced during mitochondrial respiration, this reactive oxygen species, can mediate dynamic regulation of neuronal activity and transmitter release by activating inhibitory ATP-sensitive K(+) (KATP ) channels, as well as a class of excitatory non-selective cation channels, TRPM2. Studies using ex vivo guinea pig brain slices have revealed that activity-generated H2 O2 can act via KATP channels to inhibit dopamine release in dorsal striatum and dopamine neuron activity in the substantia nigra pars compacta. In sharp contrast, endogenously generated H2 O2 enhances the excitability of GABAergic projection neurons in the dorsal striatum and substantia nigra pars reticulata by activating TRPM2 channels. These studies suggest that the balance of excitation vs. inhibition produced in a given cell by metabolically generated H2 O2 will be dictated by the relative abundance of H2 O2 -sensitive ion channel targets that receive this translational signal.

Toward sophisticated basal ganglia neuromodulation: Review on basal ganglia deep brain stimulation

This review presents state-of-the-art knowledge about the roles of the basal ganglia (BG) in action-selection, cognition, and motivation, and how this knowledge has been used to improve deep brain stimulation (DBS) treatment of neurological and psychiatric disorders. Such pathological conditions include Parkinson's disease, Huntington's disease, Tourette syndrome, depression, and obsessive-compulsive disorder. The first section presents evidence supporting current hypotheses of how the cortico-BG circuitry works to select motor and emotional actions, and how defects in this circuitry can cause symptoms of the BG diseases. Emphasis is given to the role of striatal dopamine on motor performance, motivated behaviors and learning of procedural memories. Next, the use of cutting-edge electrochemical techniques in animal and human studies of BG functioning under normal and disease conditions is discussed. Finally, functional neuroimaging studies are reviewed; these works have shown the relationship between cortico-BG structures activated during DBS and improvement of disease symptoms.

A behavioral task for investigating action discovery, selection and switching: comparison between types of reinforcer

Action discovery and selection are critical cognitive processes that are understudied at the cellular and systems neuroscience levels. Presented here is a new rodent joystick task suitable to test these processes due to the range of action possibilities that can be learnt while performing the task. Rats learned to manipulate a joystick while progressing through task milestones that required increasing degrees of movement accuracy. In a switching phase designed to measure action discovery, rats were repeatedly required to discover new target positions to meet changing task demands. Behavior was compared using both food and electrical brain stimulation reward (BSR) of the substantia nigra as reinforcement. Rats reinforced with food and those with BSR performed similarly overall, although BSR-treated rats exhibited greater vigor in responding. In the switching phase, rats learnt new actions to adapt to changing task demands, reflecting action discovery processes. Because subjects are required to learn different goal-directed actions, this task could be employed in further investigations of the cellular mechanisms of action discovery and selection. Additionally, this task could be used to assess the behavioral flexibility impairments seen in conditions such as Parkinson's disease and obsessive-compulsive disorder. The versatility of the task will enable cross-species investigations of these impairments.

Finding minimal action sequences with a simple evaluation of actions

Animals are able to discover the minimal number of actions that achieves an outcome (the minimal action sequence). In most accounts of this, actions are associated with a measure of behavior that is higher for actions that lead to the outcome with a shorter action sequence, and learning mechanisms find the actions associated with the highest measure. In this sense, previous accounts focus on more than the simple binary signal of "was the outcome achieved?"; they focus on "how well was the outcome achieved?" However, such mechanisms may not govern all types of behavioral development. In particular, in the process of action discovery (Redgrave and Gurney, 2006), actions are reinforced if they simply lead to a salient outcome because biological reinforcement signals occur too quickly to evaluate the consequences of an action beyond an indication of the outcome's occurrence. Thus, action discovery mechanisms focus on the simple evaluation of "was the outcome achieved?" and not "how well was the outcome achieved?" Notwithstanding this impoverishment of information, can the process of action discovery find the minimal action sequence? We address this question by implementing computational mechanisms, referred to in this paper as no-cost learning rules, in which each action that leads to the outcome is associated with the same measure of behavior. No-cost rules focus on "was the outcome achieved?" and are consistent with action discovery. No-cost rules discover the minimal action sequence in simulated tasks and execute it for a substantial amount of time. Extensive training, however, results in extraneous actions, suggesting that a separate process (which has been proposed in action discovery) must attenuate learning if no-cost rules participate in behavioral development. We describe how no-cost rules develop behavior, what happens when attenuation is disrupted, and relate the new mechanisms to wider computational and biological context.

Internally generated sequences in learning and executing goal-directed behavior

A network of brain structures including hippocampus (HC), prefrontal cortex, and striatum controls goal-directed behavior and decision making. However, the neural mechanisms underlying these functions are unknown. Here, we review the role of 'internally generated sequences': structured, multi-neuron firing patterns in the network that are not confined to signaling the current state or location of an agent, but are generated on the basis of internal brain dynamics. Neurophysiological studies suggest that such sequences fulfill functions in memory consolidation, augmentation of representations, internal simulation, and recombination of acquired information. Using computational modeling, we propose that internally generated sequences may be productively considered a component of goal-directed decision systems, implementing a sampling-based inference engine that optimizes goal acquisition at multiple timescales of on-line choice, action control, and learning.

Dopaminergic modulation of striatal networks in health and Parkinson's disease

In the last couple of years, there have been significant advances in our understanding of how dopamine modulates striatal circuits underlying goal-directed behaviors and how therapeutic interventions intended to normalize disordered dopaminergic signaling can go awry. This review summarizes some of the advances in this field with a translational focus on Parkinson's disease.

Dissociable effects of dopamine on learning and performance within sensorimotor striatum

Striatal dopamine is an important modulator of current behavior, as seen in the rapid and dramatic effects of dopamine replacement therapy in Parkinson Disease (PD). Yet there is also extensive evidence that dopamine acts as a learning signal, modulating synaptic plasticity within striatum to affect future behavior. Disentangling these "performance" and "learning" functions is important for designing effective, long-term PD treatments. We conducted a series of unilateral drug manipulations and dopamine terminal lesions in the dorsolateral striatum of rats highly-trained to perform brief instructed head/neck movements (two-alternative forced choice task). Reaction times and accuracy were measured longitudinally to determine if task behavior changed immediately, progressed over time, and/or persisted after drug withdrawal. Enhanced dopamine signaling with amphetamine caused an immediate, nonprogressive, and bilateral decrease in reaction times (RT). The altered RT distributions were consistent with reduced distance to threshold in the linear approach to threshold with ergodic rate (LATER) model of decision-making. Conversely, the dopamine antagonist flupenthixol caused experience-dependent, persistent changes in RT and accuracy indicative of a "learning" effect. These RT distributions were consistent with a slowed rate of approach to decision threshold. Our results show that dopaminergic signaling makes dissociable contributions to current and future behavior even within a single striatal subregion, and provide important clues for both models of normal decision-making and the design of novel drug therapies in PD.

The intralaminar thalamus-an expressway linking visual stimuli to circuits determining agency and action selection

Anatomical investigations have revealed connections between the intralaminar thalamic nuclei and areas such as the superior colliculus (SC) that receive short latency input from visual and auditory primary sensory areas. The intralaminar nuclei in turn project to the major input nucleus of the basal ganglia, the striatum, providing this nucleus with a source of subcortical excitatory input. Together with a converging input from the cerebral cortex, and a neuromodulatory dopaminergic input from the midbrain, the components previously found necessary for reinforcement learning in the basal ganglia are present. With this intralaminar sensory input, the basal ganglia are thought to play a primary role in determining what aspect of an organism's own behavior has caused salient environmental changes. Additionally, subcortical loops through thalamic and basal ganglia nuclei are proposed to play a critical role in action selection. In this mini review we will consider the anatomical and physiological evidence underlying the existence of these circuits. We will propose how the circuits interact to modulate basal ganglia output and solve common behavioral learning problems of agency determination and action selection.

Differential implication of dorsolateral and dorsomedial srtiatum in encoding and recovery processes of latent inhibition

The dorsal striatum has been ascribed to different behavioral roles. While the lateral area (dls) is implicated in habitual actions, its medial part (dms) is linked to goal expectancy. According to this model, dls function includes representation of stimulus-response associations, but not of goals. Dls function has been typically analyzed with regard to movement, and there is no data indicating whether this region could processes specific stimulus-outcome associations. To test this possibility, we analyzed the effects of dls and dms inactivation on the retrieval phase, and dms lesion on the acquisition phase of a latent inhibition procedure using two conditions, long and short presentations of the future conditioned stimulus. Contrary to current theories of basal ganglia function, we report evidence in favor of the dls involvement in cognitive processes of learning and retrieval. Moreover, we provide data about the sequential relationship between dms and dls, in which the dms could be involved, but it would not be critical, in new learning and the dls could be subsequently involved in consolidating cognitive routines.

Procedural-based category learning in patients with Parkinson's disease: impact of category number and category continuity

Previously we found that Parkinson's disease (PD) patients are impaired in procedural-based category learning when category membership is defined by a nonlinear relationship between stimulus dimensions, but these same patients are normal when the rule is defined by a linear relationship (Maddox and Filoteo, 2001; Filoteo et al., 2005a,b). We suggested that PD patients' impairment was due to a deficit in recruiting “striatal units” to represent complex nonlinear rules. In the present study, we further examined the nature of PD patients' procedural-based deficit in two experiments designed to examine the impact of (1) the number of categories, and (2) category discontinuity on learning. Results indicated that PD patients were impaired only under discontinuous category conditions but were normal when the number of categories was increased from two to four. The lack of impairment in the four-category condition suggests normal integrity of striatal medium spiny cells involved in procedural-based category learning. In contrast, and consistent with our previous observation of a nonlinear deficit, the finding that PD patients were impaired in the discontinuous condition suggests that these patients are impaired when they have to associate perceptually distinct exemplars with the same category. Theoretically, this deficit might be related to dysfunctional communication among medium spiny neurons within the striatum, particularly given that these are cholinergic neurons and a cholinergic deficiency could underlie some of PD patients' cognitive impairment.

Emergent structured transition from variation to repetition in a biologically-plausible model of learning in basal ganglia

Often, when animals encounter an unexpected sensory event, they transition from executing a variety of movements to repeating the movement(s) that may have caused the event. According to a recent theory of action discovery (Redgrave and Gurney, 2006), repetition allows the animal to represent those movements, and the outcome, as an action for later recruitment. The transition from variation to repetition often follows a non-random, structured, pattern. While the structure of the pattern can be explained by sophisticated cognitive mechanisms, simpler mechanisms based on dopaminergic modulation of basal ganglia (BG) activity are thought to underlie action discovery (Redgrave and Gurney, 2006). In this paper we ask the question: can simple BG-mediated mechanisms account for a structured transition from variation to repetition, or are more sophisticated cognitive mechanisms always necessary? To address this question, we present a computational model of BG-mediated biasing of behavior. In our model, unlike most other models of BG function, the BG biases behavior through modulation of cortical response to excitation; many possible movements are represented by the cortical area; and excitation to the cortical area is topographically-organized. We subject the model to simple reaching tasks, inspired by behavioral studies, in which a location to which to reach must be selected. Locations within a target area elicit a reinforcement signal. A structured transition from variation to repetition emerges from simple BG-mediated biasing of cortical response to excitation. We show how the structured pattern influences behavior in simple and complicated tasks. We also present analyses that describe the structured transition from variation to repetition due to BG-mediated biasing and from biasing that would be expected from a type of cognitive biasing, allowing us to compare behavior resulting from these types of biasing and make connections with future behavioral experiments.

Sensory regulation of dopaminergic cell activity: Phenomenology, circuitry and function

Dopaminergic neurons in a range of species are responsive to sensory stimuli. In the anesthetized preparation, responses to non-noxious and noxious sensory stimuli are usually tonic in nature, although long-duration changes in activity have been reported in the awake preparation as well. However, in the awake preparation, short-latency, phasic changes in activity are most common. These phasic responses can occur to unconditioned aversive and non-aversive stimuli, as well as to the stimuli which predict them. In both the anesthetized and awake preparations, not all dopaminergic neurons are responsive to sensory stimuli, however responsive neurons tend to respond to more than a single stimulus modality. Evidence suggests that short-latency sensory information is provided to dopaminergic neurons by relatively primitive subcortical structures - including the midbrain superior colliculus for vision and the mesopontine parabrachial nucleus for pain and possibly gustation. Although short-latency visual information is provided to dopaminergic neurons by the relatively primitive colliculus, dopaminergic neurons can discriminate between complex visual stimuli, an apparent paradox which can be resolved by the recently discovered route of information flow through to dopaminergic neurons from the cerebral cortex, via a relay in the colliculus. Given that projections from the cortex to the colliculus are extensive, such a relay potentially allows the activity of dopaminergic neurons to report the results of complex stimulus processing from widespread areas of the cortex. Furthermore, dopaminergic neurons could acquire their ability to reflect stimulus value by virtue of reward-related modification of sensory processing in the cortex. At the forebrain level, sensory-related changes in the tonic activity of dopaminergic neurons may regulate the impact of the cortex on forebrain structures such as the nucleus accumbens. In contrast, the short latency of the phasic responses to sensory stimuli in dopaminergic neurons, coupled with the activation of these neurons by non-rewarding stimuli, suggests that phasic responses of dopaminergic neurons may provide a signal to the forebrain which indicates that a salient event has occurred (and possibly an estimate of how salient that event is). A stimulus-related salience signal could be used by downstream systems to reinforce behavioral choices.

Habituation of reinforcer effectiveness

In this paper we propose an integrative model of habituation of reinforcer effectiveness (HRE) that links behavioral- and neural-based explanations of reinforcement. We argue that HRE is a fundamental property of reinforcing stimuli. Most reinforcement models implicitly suggest that the effectiveness of a reinforcer is stable across repeated presentations. In contrast, an HRE approach predicts decreased effectiveness due to repeated presentation. We argue that repeated presentation of reinforcing stimuli decreases their effectiveness and that these decreases are described by the behavioral characteristics of habituation (McSweeney and Murphy, 2009; Rankin etal., 2009). We describe a neural model that postulates a positive association between dopamine neurotransmission and HRE. We present evidence that stimulant drugs, which artificially increase dopamine neurotransmission, disrupt (slow) normally occurring HRE and also provide evidence that stimulant drugs have differential effects on operant responding maintained by reinforcers with rapid vs. slow HRE rates. We hypothesize that abnormal HRE due to genetic and/or environmental factors may underlie some behavioral disorders. For example, recent research indicates that slow-HRE is predictive of obesity. In contrast ADHD may reflect “accelerated-HRE.” Consideration of HRE is important for the development of effective reinforcement-based treatments. Finally, we point out that most of the reinforcing stimuli that regulate daily behavior are non-consumable environmental/social reinforcers which have rapid-HRE. The almost exclusive use of consumable reinforcers with slow-HRE in pre-clinical studies with animals may have caused the importance of HRE to be overlooked. Further study of reinforcing stimuli with rapid-HRE is needed in order to understand how habituation and reinforcement interact and regulate behavior.

Dopaminergic basis of salience dysregulation in psychosis

Disrupted salience processing is proposed as central in linking dysregulated dopamine function with psychotic symptoms. Several strands of evidence are now converging in support of this model. Animal studies show that midbrain dopamine neurons are activated by unexpected salient events. In psychotic patients, neurochemical studies have confirmed subcortical striatal dysregulation of dopaminergic neurotransmission, whereas functional magnetic resonance imaging (fMRI) studies of salience tasks have located alterations in prefrontal and striatal dopaminergic projection fields. At the clinical level, this may account for the altered sense of meaning and significance that predates the onset of psychosis. This review draws these different strands of evidence together in support of an emerging understanding of how dopamine dysregulation may lead to aberrant salience and psychotic symptoms.

Sensory aspects of movement disorders

Movement disorders, which include disorders such as Parkinson's disease, dystonia, Tourette's syndrome, restless legs syndrome, and akathisia, have traditionally been considered to be disorders of impaired motor control resulting predominantly from dysfunction of the basal ganglia. This notion has been revised largely because of increasing recognition of associated behavioural, psychiatric, autonomic, and other non-motor symptoms. The sensory aspects of movement disorders include intrinsic sensory abnormalities and the effects of external sensory input on the underlying motor abnormality. The basal ganglia, cerebellum, thalamus, and their connections, coupled with altered sensory input, seem to play a key part in abnormal sensorimotor integration. However, more investigation into the phenomenology and physiological basis of sensory abnormalities, and about the role of the basal ganglia, cerebellum, and related structures in somatosensory processing, and its effect on motor control, is needed.

Mouse models of neurodevelopmental disease of the basal ganglia and associated circuits

This chapter focuses on neurodevelopmental diseases that are tightly linked to abnormal function of the striatum and connected structures. We begin with an overview of three representative diseases in which striatal dysfunction plays a key role--Tourette syndrome and obsessive-compulsive disorder, Rett's syndrome, and primary dystonia. These diseases highlight distinct etiologies that disrupt striatal integrity and function during development, and showcase the varied clinical manifestations of striatal dysfunction. We then review striatal organization and function, including evidence for striatal roles in online motor control/action selection, reinforcement learning, habit formation, and action sequencing. A key barrier to progress has been the relative lack of animal models of these diseases, though recently there has been considerable progress. We review these efforts, including their relative merits providing insight into disease pathogenesis, disease symptomatology, and basal ganglia function.

Cortical regulation of dopaminergic neurons: role of the midbrain superior colliculus

Dopaminergic (DA) neurons respond to stimuli in a wide range of modalities, although the origin of the afferent sensory signals has only recently begun to emerge. In the case of vision, an important source of short-latency sensory information seems to be the midbrain superior colliculus (SC). However, longer-latency responses have been identified that are less compatible with the primitive perceptual capacities of the colliculus. Rather, they seem more in keeping with the processing capabilities of the cortex. Given that there are robust projections from the cortex to the SC, we examined whether cortical information could reach DA neurons via a relay in the colliculus. The somatosensory barrel cortex was stimulated electrically in the anesthetized rat with either single pulses or pulse trains. Although single pulses produced small phasic activations in the colliculus, they did not elicit responses in the majority of DA neurons. However, after disinhibitory intracollicular injections of the GABA_A antagonist bicuculline, collicular responses were substantially enhanced and previously unresponsive DA neurons now exhibited phasic excitations or inhibitions. Pulse trains applied to the cortex led to phasic changes (excitations to inhibitions) in the activity of DA neurons at baseline. These were blocked or attenuated by intracollicular administration of the GABA_A agonist muscimol. Taken together, the results indicate that the cortex can communicate with DA neurons via a relay in the SC. As a consequence, DA neuronal activity reflecting the unexpected occurrence of salient events and that signaling more complex stimulus properties may have a common origin.

The endophenotype and the phenotype: temporal discrimination and adult-onset dystonia

The pathogenesis and the genetic basis of adult-onset primary torsion dystonia remain poorly understood. Because of markedly reduced penetrance in this disorder, a number of endophenotypes have been proposed; many of these may be epiphenomena secondary to disease manifestation. Mediational endophenotypes represent gene expression; the study of trait (endophenotypic) rather than state (phenotypic) characteristics avoids the misattribution of secondary adaptive cerebral changes to pathogenesis. We argue that abnormal temporal discrimination is a mediational endophenotype; its use facilitates examination of the effects of age, gender, and environment on disease penetrance in adult-onset dystonia. Using abnormal temporal discrimination in unaffected first-degree relatives as a marker for gene mutation carriage may inform exome sequencing techniques in families with few affected individuals. We further hypothesize that abnormal temporal discrimination reflects dysfunction in an evolutionarily conserved subcortical-basal ganglia circuit for the detection of salient novel environmental change. The mechanisms of dysfunction in this pathway should be a focus for future research in the pathogenesis of adult-onset primary torsion dystonia.

Nicotinic receptors regulate the dynamic range of dopamine release in vivo

Nicotinic acetylcholine receptors (nAChRs) are expressed presynaptically on dopamine axon terminals, and their activation by endogenous acetylcholine from striatal cholinergic interneurons enhances dopamine release both independently of and in concert with dopamine neuron activity. Acute nAChR inactivation is believed to enhance the contrast between low- and high-frequency dopamine cell activity. Although these studies reveal a key role for acute activation and inactivation of nAChRs in striatal microcircuitry, it remains unknown if chronic inactivation/desensitization of nAChRs can alter dopamine release dynamics. Using in vivo cyclic voltammetry in anaesthetized mice, we examined whether chronic inactivation of nAChRs modulates dopamine release across a parametric range of stimulation, varying both frequency and pulse number. Deletion of β2*nAChRs and chronic nicotine exposure greatly diminished dopamine release across the entire range of stimulation parameters. In addition, we observed a facilitation of dopamine release at low frequency and pulse number in wild-type mice that is absent in the β2* knockout and chronic nicotine mice. These data suggest that deletion or chronic desensitization of nAChRs reduces the dynamic range of dopamine release in response to dopamine cell activity, decreasing rather than increasing contrast between high and low dopamine activity.

Neural representation of a target auditory memory in a cortico-basal ganglia pathway

Vocal learning in songbirds, like speech acquisition in humans, entails a period of sensorimotor integration during which vocalizations are evaluated via auditory feedback and progressively refined to achieve an imitation of memorized vocal sounds. This process requires the brain to compare feedback of current vocal behavior to a memory of target vocal sounds. We report the discovery of two distinct populations of neurons in a cortico-basal ganglia circuit of juvenile songbirds (zebra finches, Taeniopygia guttata) during vocal learning: (1) one in which neurons are selectively tuned to memorized sounds and (2) another in which neurons are selectively tuned to self-produced vocalizations. These results suggest that neurons tuned to learned vocal sounds encode a memory of those target sounds, whereas neurons tuned to self-produced vocalizations encode a representation of current vocal sounds. The presence of neurons tuned to memorized sounds is limited to early stages of sensorimotor integration: after learning, the incidence of neurons encoding memorized vocal sounds was greatly diminished. In contrast to this circuit, neurons known to drive vocal behavior through a parallel cortico-basal ganglia pathway show little selective tuning until late in learning. One interpretation of these data is that representations of current and target vocal sounds in the shell circuit are used to compare ongoing patterns of vocal feedback to memorized sounds, whereas the parallel core circuit has a motor-related role in learning. Such a functional subdivision is similar to mammalian cortico-basal ganglia pathways in which associative-limbic circuits mediate goal-directed responses, whereas sensorimotor circuits support motor aspects of learning.

Punishing unfairness: rewarding or the organization of a reactively aggressive response?

OBJECTIVES

The neural correlates of human cooperative behavior remain poorly understood. Previous work has suggested that increases in striatal activation while punishing unfair offers represents reward signaling. However, other regions are also implicated when punishing others, for example dorsomedial frontal cortex (dmFC), anterior insula cortex (AIC), and periaqueductal gray (PAG). Moreover, the response of other regions implicated in signaling reward, for example ventromedial prefrontal cortex (vmPFC) or posterior cingulate cortex (PCC), has not been systematically examined.

EXPERIMENTAL DESIGN

Functional magnetic resonance imaging utilizing parametric modulation was conducted on 21 healthy adults participating in a social exchange paradigm.

PRINCIPAL OBSERVATIONS

Participants showed significant positive modulation of activity as a function of delivered punishment in caudate, dmFC, AIC, and PAG; that is, higher punishments by participants of unsatisfactory offers were associated with increasing activity within these regions. However, participants showed significant negative modulation of activity as a function of delivered punishment in vmPFC and PCC; increases in punishment level by participants were associated with decreases in activity within these regions.

CONCLUSIONS

The current data question whether caudate activity when punishing unfair offers should be considered to indicate the reward value of this punishment. Instead, this activity, in conjunction with activity within dmFC, AIC, and PAG, may represent the organization of an untypical, punishing response that represents a reactive aggressive response to provocation. Notably, an inverse, regulatory relationship between vmPFC and PAG activity has been previously implicated in the context of another stimulus for reactive aggression; looming threat (Mobbs et al. [2007]: Science 317:1079-1083).

Vision, reanimated and reimagined

The publication in 1982 of David Marr's Vision has delivered a singular boost and a course correction to the science of vision. Thirty years later, cognitive science is being transformed by the new ways of thinking about what it is that the brain computes, how it does that, and, most importantly, why cognition requires these computations and not others. This ongoing process still owes much of its impetus and direction to the sound methodology, engaging style, and unique voice of Marr's Vision.

Unilateral skill acquisition induces bilateral NMDA receptor subunit composition shifts in the rat sensorimotor striatum

The sensorimotor striatum is critical for the acquisition and consolidation of skilled learning-related motor sequences. Excitatory corticostriatal synapses undergo neuroplastic changes that impact signal transmission efficacy. Modification of N-methyl d-aspartate (NMDA) and α-amino-3-hydroxyl-5-methyl-4-isoxazolepropionic acid (AMPA) glutamate receptor subunit composition and phosphorylation is critical for bidirectional experience-driven plasticity observed at these synapses. Metaplastic regulation of the ratio of NR2A to NR2B subunits of the NMDA receptor controls the threshold for the induction of subsequent plasticity. However, little is known about how repeated practice effects the differential regulation of glutamate receptors during the acquisition of a unilateral motor skill. Using immunoblot analysis, we assessed changes in NMDA and AMPA receptors during the associative stage of skill acquisition in synaptoneurosome preparations from the rat sensorimotor striatum. We found that the NR2A/B subunit ratio in the striatum contralateral to the trained limb decreased during skill acquisition optimizing the threshold for inducing subsequent synaptic plasticity during learning of the lateralized motor skill. In contrast, there was a significant increase in the NR2A/B subunit ratio in the ipsilateral striatum making the induction of subsequent plasticity more difficult. In addition, there was a selective decrease in AMPAR phosphorylation levels at serine site 831 but not 845 on the GluR1 subunit ipsilaterally with a trend toward a decrease contralaterally. These findings suggest that the successful acquisition of a lateralized motor skill necessitates the integration of motor programs in both striata, each of which reflects unique changes in the NR2A/B ratio that modulate the different task demands on the associated limb.

Updating of action-outcome associations is prevented by inactivation of the posterior pedunculopontine tegmental nucleus

•

The pedunculopontine tegmental nucleus is essential for action–outcome learning.

•

Sensitivity to instrumental contingency degradation is blocked by PPTg inactivation.

•

Inactivation of PPTg does not change performance of previously learnt operant tasks.

•

This is the first demonstration of a role for brainstem in action–outcome learning.

•

Learning functions of basal ganglia extend into the deepest parts of the circuitry.

The pedunculopontine tegmental nucleus (PPTg) is in a pivotal position between the basal ganglia and brainstem: it is able to influence and regulate all levels of basal ganglia and corticostriatal activity as well as being a key component of brainstem reticular and motor control circuitry. Consistent with its anatomical position, the PPTg has previously been shown to process rapid, salient sensory input, is a target for Parkinson’s disease treatments and has been implicated in associative learning. We explicitly investigated the role of the posterior pPPTg (pPPTg) in action–outcome processes, where actions are performed with the goal-directed aim of obtaining an anticipated outcome. We assessed rats’ sensitivity to degradation of the contingency between actions (lever pressing) and outcomes (food reward) during either inactivation of pPPTg by microinjection of the GABA agonist muscimol or control infusions of saline. In response to the degradation of contingency between lever press and food reward, saline treated rats rapidly reduced rates of lever pressing whereas muscimol treated rats (pPPTg inactivation) maintained previous lever pressing rates. In contrast, when the contingency between lever press and food reward was unchanged saline and muscimol treated rats maintained their previous rates of lever pressing. This shows that the pPPTg is critically required for updating associations between actions and outcomes, but not in the continued performance of previously learned associations. These results are consistent with a role for the PPTg in ‘higher-order’ associative learning and are the first to demonstrate a brainstem role in action–outcome learning.

A biologically plausible embodied model of action discovery

During development, animals can spontaneously discover action-outcome pairings enabling subsequent achievement of their goals. We present a biologically plausible embodied model addressing key aspects of this process. The biomimetic model core comprises the basal ganglia and its loops through cortex and thalamus. We incorporate reinforcement learning (RL) with phasic dopamine supplying a sensory prediction error, signalling "surprising" outcomes. Phasic dopamine is used in a cortico-striatal learning rule which is consistent with recent data. We also hypothesized that objects associated with surprising outcomes acquire "novelty salience" contingent on the predicability of the outcome. To test this idea we used a simple model of prediction governing the dynamics of novelty salience and phasic dopamine. The task of the virtual robotic agent mimicked an in vivo counterpart (Gancarz et al., 2011) and involved interaction with a target object which caused a light flash, or a control object which did not. Learning took place according to two schedules. In one, the phasic outcome was delivered after interaction with the target in an unpredictable way which emulated the in vivo protocol. Without novelty salience, the model was unable to account for the experimental data. In the other schedule, the phasic outcome was reliably delivered and the agent showed a rapid increase in the number of interactions with the target which then decreased over subsequent sessions. We argue this is precisely the kind of change in behavior required to repeatedly present representations of context, action and outcome, to neural networks responsible for learning action-outcome contingency. The model also showed cortico-striatal plasticity consistent with learning a new action in basal ganglia. We conclude that action learning is underpinned by a complex interplay of plasticity and stimulus salience, and that our model contains many of the elements for biological action discovery to take place.